[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:42 dist-test-slave-wpdm krb5kdc[9059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:42 dist-test-slave-wpdm krb5kdc[9059](info): set up 2 sockets
Sep 25 05:06:42 dist-test-slave-wpdm krb5kdc[9059](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 05:06:43.078642 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:06:43 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:06:43.253278 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:45829
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:45829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:43.579928 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:43.580291 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:43.580473 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:43.588080 9072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:43.588176 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:43.588284 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:43.588382 9072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:43.588466 9072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:43.598356 9072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:45829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:45829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:43.599735 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:43.601034 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:43.613293 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:43.613318 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:43.613260 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:43.613957 9072 server_base.cc:1034] running on GCE node
I20260925 05:06:43.614756 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:43.616079 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:43.617274 9072 hybrid_clock.cc:648] HybridClock initialized: now 1790312803617234 us; error 36 us; skew 500 ppm
I20260925 05:06:43.617609 9072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:43 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:43.620808 9072 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:06:43.622304 9072 webserver.cc:460] Webserver started at http://127.8.215.62:36913/ using document root <none> and password file <none>
I20260925 05:06:43.622967 9072 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:43.623102 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:43.623401 9072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:43.626356 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "5dcd5232416f4a0e9bde2bba1467ff27"
format_stamp: "Formatted at 2026-09-25 05:06:43 on dist-test-slave-wpdm"
server_key: "40f368c799f93170001ef38c31ef8b78"
server_key_iv: "f7063e7f56dbbbc9193c55a87c14af96"
server_key_version: "encryptionkey@0"
I20260925 05:06:43.627141 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "5dcd5232416f4a0e9bde2bba1467ff27"
format_stamp: "Formatted at 2026-09-25 05:06:43 on dist-test-slave-wpdm"
server_key: "40f368c799f93170001ef38c31ef8b78"
server_key_iv: "f7063e7f56dbbbc9193c55a87c14af96"
server_key_version: "encryptionkey@0"
I20260925 05:06:43.632200 9072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:06:43.635521 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:43.636718 9072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:06:43.637009 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "5dcd5232416f4a0e9bde2bba1467ff27"
format_stamp: "Formatted at 2026-09-25 05:06:43 on dist-test-slave-wpdm"
server_key: "40f368c799f93170001ef38c31ef8b78"
server_key_iv: "f7063e7f56dbbbc9193c55a87c14af96"
server_key_version: "encryptionkey@0"
I20260925 05:06:43.637200 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:43.657722 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:43.661901 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:43.662196 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:43.681687 9072 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:45829
I20260925 05:06:43.681762 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:45829 every 8 connection(s)
I20260925 05:06:43.683394 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:43.687318 9139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:43.689678 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9072
I20260925 05:06:43.689967 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:06:43.691763 9052 external_mini_cluster.cc:1428] Setting key 6ad942edb3d31b5a2a34d9a61bc5a152
I20260925 05:06:43.697582 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27: Bootstrap starting.
I20260925 05:06:43.700475 9139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:43.701520 9139 log.cc:824] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:43.704363 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27: No bootstrap required, opened a new log
I20260925 05:06:43.714743 9139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45829 } }
I20260925 05:06:43.715322 9139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:43.715435 9139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcd5232416f4a0e9bde2bba1467ff27, State: Initialized, Role: FOLLOWER
I20260925 05:06:43.716143 9139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45829 } }
I20260925 05:06:43.716428 9139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:43.716596 9139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:43.716789 9139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:43.719295 9139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45829 } }
I20260925 05:06:43.719839 9139 leader_election.cc:302] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dcd5232416f4a0e9bde2bba1467ff27; no voters:
I20260925 05:06:43.720551 9139 leader_election.cc:288] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:43.720752 9144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 1 FOLLOWER]: Leader election won for term 1
Sep 25 05:06:43 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:43.723104 9139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:43.723591 9144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcd5232416f4a0e9bde2bba1467ff27, State: Running, Role: LEADER
I20260925 05:06:43.724212 9144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45829 } }
I20260925 05:06:43.731472 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45829 } } }
I20260925 05:06:43.731873 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:43.732364 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dcd5232416f4a0e9bde2bba1467ff27. Latest consensus state: current_term: 1 leader_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcd5232416f4a0e9bde2bba1467ff27" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45829 } } }
I20260925 05:06:43.732647 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:43.736305 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:43.695645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.8.215.62:45829)
0925 05:06:43.696533 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:43.696549 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:43.698061 (+ 1512us) server_negotiation.cc:407] Connection header received
0925 05:06:43.707055 (+ 8994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:43.707132 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:43.707716 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:43.708208 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:43.711051 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:43.713188 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:43.714897 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:43.715991 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:43.716183 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:43.723395 (+ 7212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:43.723413 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:43.723440 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:43.723572 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:43.728657 (+ 5085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:43.729957 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:43.729970 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:43.729982 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:43.730284 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:43.730956 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:43.730962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:43.730967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:43.731905 (+ 938us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:43.733792 (+ 1887us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:43.734115 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 05:06:43.734651 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":427,"threads_started":1}
I20260925 05:06:43.738440 9153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:43.743213 9153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:43.751506 9153 catalog_manager.cc:1347] Generated new cluster ID: a88c7482e61e4e67952372ba2ea0aad7
I20260925 05:06:43.751644 9153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:43.759521 9153 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:43.760860 9153 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:43.771538 9153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dcd5232416f4a0e9bde2bba1467ff27: Generated new TSK 0
I20260925 05:06:43.772365 9153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:06:43.826821 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45829
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:44.154896 9165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:44.155171 9165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:44.155341 9165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:44.162510 9165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:44.162649 9165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:44.162823 9165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:06:44.172448 9165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:44.173784 9165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:44.175192 9165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:44.188159 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:44.188172 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:44.188227 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:44.189435 9165 server_base.cc:1034] running on GCE node
I20260925 05:06:44.190052 9165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:44.190973 9165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:44.192196 9165 hybrid_clock.cc:648] HybridClock initialized: now 1790312804192145 us; error 68 us; skew 500 ppm
I20260925 05:06:44.192497 9165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:44 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312804, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:44.195657 9165 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:06:44.197118 9165 webserver.cc:460] Webserver started at http://127.8.215.1:36921/ using document root <none> and password file <none>
I20260925 05:06:44.197782 9165 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:44.197921 9165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:44.198221 9165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:44.201133 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "9623db87fb2948a3870d5f1b2e38df8b"
format_stamp: "Formatted at 2026-09-25 05:06:44 on dist-test-slave-wpdm"
server_key: "514f765e98898b226a99a65da4a6d63f"
server_key_iv: "e81569bd9000a9457874e66ddbe97178"
server_key_version: "encryptionkey@0"
I20260925 05:06:44.201890 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "9623db87fb2948a3870d5f1b2e38df8b"
format_stamp: "Formatted at 2026-09-25 05:06:44 on dist-test-slave-wpdm"
server_key: "514f765e98898b226a99a65da4a6d63f"
server_key_iv: "e81569bd9000a9457874e66ddbe97178"
server_key_version: "encryptionkey@0"
I20260925 05:06:44.206707 9165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:06:44.210083 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:44.211398 9165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:06:44.211572 9165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "9623db87fb2948a3870d5f1b2e38df8b"
format_stamp: "Formatted at 2026-09-25 05:06:44 on dist-test-slave-wpdm"
server_key: "514f765e98898b226a99a65da4a6d63f"
server_key_iv: "e81569bd9000a9457874e66ddbe97178"
server_key_version: "encryptionkey@0"
I20260925 05:06:44.211760 9165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:44.231348 9165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:44.235653 9165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:44.235947 9165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:44.237010 9165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:44.238551 9165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:44.238651 9165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:44.238794 9165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:44.238865 9165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:44.273942 9165 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:39919
I20260925 05:06:44.273947 9292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:39919 every 8 connection(s)
I20260925 05:06:44.275302 9165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:44.284202 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9165
I20260925 05:06:44.284423 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:44.285193 9052 external_mini_cluster.cc:1428] Setting key 7b655c74b2a3a10840b38c778e8cfc15
Sep 25 05:06:44 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312804, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:44.296850 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:44.277914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:60981 (local address 127.8.215.62:45829)
0925 05:06:44.278359 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:44.278365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:44.279463 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 05:06:44.281333 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:44.281337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:44.281410 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:44.281508 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:44.282918 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.283646 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.286013 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.286511 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.286610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:44.290529 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:44.290539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:44.290546 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:44.290591 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:44.293402 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:44.294349 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:44.294353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:44.294355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:44.294464 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:44.295022 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:44.295027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:44.295029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:44.295306 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:44.295426 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:44.296323 (+ 897us) server_negotiation.cc:299] Negotiation successful
0925 05:06:44.296494 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":323,"thread_start_us":231,"threads_started":1}
I20260925 05:06:44.298336 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:44.278571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:45829 (local address 127.8.215.1:60981)
0925 05:06:44.279299 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:44.279341 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:44.281080 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:44.281713 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:44.281721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:44.282216 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:44.282745 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:44.282761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.283813 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.283820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:44.285872 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:44.285883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.286734 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.286741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:44.287002 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:44.287793 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:44.287819 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:44.290288 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:44.293537 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:44.293550 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:44.293570 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:44.294214 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:44.294564 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:44.294572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:44.294578 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:44.294814 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:44.295448 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:44.295458 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:44.295902 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 05:06:44.296210 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 05:06:44.296813 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":319,"threads_started":1}
I20260925 05:06:44.300140 9293 heartbeater.cc:347] Connected to a master server at 127.8.215.62:45829
I20260925 05:06:44.300441 9293 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:44.301318 9293 heartbeater.cc:511] Master 127.8.215.62:45829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
I20260925 05:06:44.303561 9104 ts_manager.cc:194] Registered new tserver with Master: 9623db87fb2948a3870d5f1b2e38df8b (127.8.215.1:39919)
I20260925 05:06:44.305040 9104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:60981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:06:44.326087 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45829
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:44.673938 9299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:44.674343 9299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:44.674541 9299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:44.682085 9299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:44.682261 9299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:44.682471 9299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:06:44.696038 9299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:44.697875 9299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:44.699555 9299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:44.714771 9305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:44.714866 9304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:44.716233 9299 server_base.cc:1034] running on GCE node
W20260925 05:06:44.716878 9307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:44.717582 9299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:44.718612 9299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:44.719848 9299 hybrid_clock.cc:648] HybridClock initialized: now 1790312804719768 us; error 53 us; skew 500 ppm
I20260925 05:06:44.720206 9299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:44 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312804, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:44.723666 9299 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:06:44.725375 9299 webserver.cc:460] Webserver started at http://127.8.215.2:39679/ using document root <none> and password file <none>
I20260925 05:06:44.726241 9299 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:44.726348 9299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:44.726612 9299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:44.729648 9299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "1af93c1fe49d48e491a976717812601a"
format_stamp: "Formatted at 2026-09-25 05:06:44 on dist-test-slave-wpdm"
server_key: "e44aa2a7b3a8d4134b89ea547b728302"
server_key_iv: "d1063ce181b0bccbaea408e339923d95"
server_key_version: "encryptionkey@0"
I20260925 05:06:44.730376 9299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "1af93c1fe49d48e491a976717812601a"
format_stamp: "Formatted at 2026-09-25 05:06:44 on dist-test-slave-wpdm"
server_key: "e44aa2a7b3a8d4134b89ea547b728302"
server_key_iv: "d1063ce181b0bccbaea408e339923d95"
server_key_version: "encryptionkey@0"
I20260925 05:06:44.735750 9299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:06:44.739202 9314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:44.740661 9299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:06:44.740913 9299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "1af93c1fe49d48e491a976717812601a"
format_stamp: "Formatted at 2026-09-25 05:06:44 on dist-test-slave-wpdm"
server_key: "e44aa2a7b3a8d4134b89ea547b728302"
server_key_iv: "d1063ce181b0bccbaea408e339923d95"
server_key_version: "encryptionkey@0"
I20260925 05:06:44.741155 9299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:44.762223 9299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:44.766342 9299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:44.766628 9299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:44.767597 9299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:44.768946 9299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:44.769018 9299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:44.769205 9299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:44.769258 9299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:44.804148 9299 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:43829
I20260925 05:06:44.804227 9426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:43829 every 8 connection(s)
I20260925 05:06:44.805472 9299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:44.813297 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9299
I20260925 05:06:44.813506 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:44.813970 9052 external_mini_cluster.cc:1428] Setting key ce60888d9982fe3961a3c07e5158a928
Sep 25 05:06:44 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312804, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:44.827018 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:44.807978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:52263 (local address 127.8.215.62:45829)
0925 05:06:44.808430 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:44.808435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:44.809401 (+ 966us) server_negotiation.cc:407] Connection header received
0925 05:06:44.811276 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:44.811282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:44.811372 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:44.811490 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:44.813109 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.813868 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.816183 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.816891 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.817041 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:44.820846 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:44.820852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:44.820855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:44.820885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:44.823580 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:44.824533 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:44.824537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:44.824540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:44.824680 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:44.825227 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:44.825230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:44.825232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:44.825472 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:44.825587 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:44.826537 (+ 950us) server_negotiation.cc:299] Negotiation successful
0925 05:06:44.826755 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":257,"threads_started":1}
I20260925 05:06:44.828809 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:44.808466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:45829 (local address 127.8.215.2:52263)
0925 05:06:44.809248 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:44.809287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:44.810997 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:44.811735 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:44.811743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:44.812249 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:44.812892 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:44.812911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.814028 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.814034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:44.816051 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:44.816062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:44.817091 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:44.817096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:44.817299 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:44.818035 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:44.818062 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:44.820542 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:44.823723 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:44.823758 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:44.823779 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:44.824427 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:44.824842 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:44.824849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:44.824856 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:44.825111 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:44.825621 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:44.825632 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:44.826098 (+ 466us) client_negotiation.cc:769] Sending connection context
0925 05:06:44.826446 (+ 348us) client_negotiation.cc:240] Negotiation successful
0925 05:06:44.827069 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":329,"threads_started":1}
I20260925 05:06:44.830307 9427 heartbeater.cc:347] Connected to a master server at 127.8.215.62:45829
I20260925 05:06:44.830647 9427 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
I20260925 05:06:44.831496 9427 heartbeater.cc:511] Master 127.8.215.62:45829 requested a full tablet report, sending...
I20260925 05:06:44.833086 9104 ts_manager.cc:194] Registered new tserver with Master: 1af93c1fe49d48e491a976717812601a (127.8.215.2:43829)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
I20260925 05:06:44.834159 9104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:52263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:06:44.854447 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45829
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:45.189194 9433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:45.189517 9433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:45.189734 9433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:45.198158 9433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:45.198366 9433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:45.198582 9433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:06:45.209856 9433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:45.211313 9433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:45.212805 9433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:45.228117 9439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:45.228263 9438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:45.228288 9441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:45.229421 9433 server_base.cc:1034] running on GCE node
I20260925 05:06:45.230019 9433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:45.230984 9433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:45.232220 9433 hybrid_clock.cc:648] HybridClock initialized: now 1790312805232171 us; error 41 us; skew 500 ppm
I20260925 05:06:45.232544 9433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:45.235980 9433 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:06:45.237588 9433 webserver.cc:460] Webserver started at http://127.8.215.3:37805/ using document root <none> and password file <none>
I20260925 05:06:45.238319 9433 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:45.238466 9433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:45.238804 9433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:45.241979 9433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "4f4db1d0f2994ccf861fe763de73eee6"
format_stamp: "Formatted at 2026-09-25 05:06:45 on dist-test-slave-wpdm"
server_key: "cd46750781a486f649b48d4b8719cf61"
server_key_iv: "6a03b340832bb0b6dd867f4143addf87"
server_key_version: "encryptionkey@0"
I20260925 05:06:45.242717 9433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "4f4db1d0f2994ccf861fe763de73eee6"
format_stamp: "Formatted at 2026-09-25 05:06:45 on dist-test-slave-wpdm"
server_key: "cd46750781a486f649b48d4b8719cf61"
server_key_iv: "6a03b340832bb0b6dd867f4143addf87"
server_key_version: "encryptionkey@0"
I20260925 05:06:45.248440 9433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 05:06:45.252276 9448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:45.253893 9433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:45.254081 9433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "4f4db1d0f2994ccf861fe763de73eee6"
format_stamp: "Formatted at 2026-09-25 05:06:45 on dist-test-slave-wpdm"
server_key: "cd46750781a486f649b48d4b8719cf61"
server_key_iv: "6a03b340832bb0b6dd867f4143addf87"
server_key_version: "encryptionkey@0"
I20260925 05:06:45.254314 9433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:45.278416 9433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:45.283322 9433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:45.283583 9433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:45.284523 9433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:45.285933 9433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:45.286008 9433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:45.286211 9433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:45.286260 9433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:45.308167 9293 heartbeater.cc:503] Master 127.8.215.62:45829 was elected leader, sending a full tablet report...
I20260925 05:06:45.325016 9433 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:35723
I20260925 05:06:45.325114 9560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:35723 every 8 connection(s)
I20260925 05:06:45.326460 9433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:45.332075 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9433
I20260925 05:06:45.332300 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:45.332890 9052 external_mini_cluster.cc:1428] Setting key e76c5f2dab8eacdc639ea761ad33e54b
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:45.350373 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.329460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:35139 (local address 127.8.215.62:45829)
0925 05:06:45.329962 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:45.329966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:45.331026 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 05:06:45.334044 (+ 3018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:45.334050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:45.334149 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:45.334270 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:45.336620 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.337639 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.339329 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.339713 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.339814 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.343836 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:45.343842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:45.343845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:45.343878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:45.346711 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.347552 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.347555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.347558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.347666 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.348299 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.348304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.348307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.348667 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:45.348850 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:45.349912 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0925 05:06:45.350101 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260925 05:06:45.351668 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.330027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:45829 (local address 127.8.215.3:35139)
0925 05:06:45.330824 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:45.330868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:45.333610 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:45.334589 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:45.334599 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:45.335513 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:45.336280 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:45.336300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.337805 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.337809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:45.339187 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:45.339199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.339991 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.339999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:45.340278 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.341055 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:45.341100 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:45.343612 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:45.346856 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:45.346864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:45.346878 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:45.347423 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:45.347787 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:45.347793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:45.347797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:45.348146 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:45.348867 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:45.348876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:45.349443 (+ 567us) client_negotiation.cc:769] Sending connection context
0925 05:06:45.349809 (+ 366us) client_negotiation.cc:240] Negotiation successful
0925 05:06:45.350361 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":351,"threads_started":1}
I20260925 05:06:45.353173 9561 heartbeater.cc:347] Connected to a master server at 127.8.215.62:45829
I20260925 05:06:45.353446 9561 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:45.354180 9561 heartbeater.cc:511] Master 127.8.215.62:45829 requested a full tablet report, sending...
I20260925 05:06:45.355578 9104 ts_manager.cc:194] Registered new tserver with Master: 4f4db1d0f2994ccf861fe763de73eee6 (127.8.215.3:35723)
I20260925 05:06:45.356410 9104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:35139
I20260925 05:06:45.362422 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:06:45.409824 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.393867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.8.215.1:39919)
0925 05:06:45.394665 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:45.394685 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:45.394720 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:06:45.394899 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:45.394907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:45.395125 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:45.395294 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:45.396864 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.397886 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.399216 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.399801 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.399948 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.404030 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:45.404043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:45.404060 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:45.404095 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:45.406933 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.407649 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.407657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.407659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.407787 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.408193 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.408200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.408206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.408548 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:45.408901 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:45.409183 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 05:06:45.409451 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":457,"threads_started":1}
W20260925 05:06:45.411303 9208 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44114
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:06:45.458245 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.442508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.8.215.1:39919)
0925 05:06:45.442772 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:45.442778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:45.443003 (+ 225us) server_negotiation.cc:407] Connection header received
0925 05:06:45.443164 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:45.443168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:45.443259 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:45.443344 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:45.445192 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.445988 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.447433 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.447924 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.448088 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.452650 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:45.452655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:45.452659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:45.452704 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:45.455232 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.455945 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.455950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.455954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.456099 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.456516 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.456521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.456525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.456940 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:45.457203 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:45.457638 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 05:06:45.457914 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 05:06:45.459952 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9165
I20260925 05:06:45.473340 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.461154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.8.215.1:39919)
0925 05:06:45.461371 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:45.461377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:45.461505 (+ 128us) server_negotiation.cc:407] Connection header received
0925 05:06:45.461630 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:45.461634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:45.461707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:45.461785 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:45.462939 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.463633 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.464971 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.465380 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.465474 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.467838 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:45.467843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:45.467847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:45.467880 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:45.470654 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.471312 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.471316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.471318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.471453 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.471992 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.471996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.471998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.472249 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:45.472426 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:45.472813 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 05:06:45.472979 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:06:45.474367 9288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:45.561481 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9165
I20260925 05:06:45.578488 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9299
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:06:45.593472 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.579404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.8.215.2:43829)
0925 05:06:45.579925 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:45.579932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:45.579959 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:06:45.580016 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:45.580030 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:45.580245 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:45.580378 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:45.581570 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.582538 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.583732 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.584254 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.584347 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.588257 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:45.588268 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:45.588282 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:45.588313 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:45.591027 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.591571 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.591577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.591580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.591706 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.592006 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.592010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.592012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.592333 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:45.592552 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:45.592884 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 05:06:45.593074 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":278,"threads_started":1}
I20260925 05:06:45.594694 9422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:45.676512 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9299
I20260925 05:06:45.704221 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9433
Sep 25 05:06:45 dist-test-slave-wpdm krb5kdc[9059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:06:45.720337 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:45.705328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.8.215.3:35723)
0925 05:06:45.705965 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:45.705971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:45.705998 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:06:45.706054 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:45.706060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:45.706276 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:45.706457 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:45.707687 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.708674 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.709999 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:45.710559 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:45.710714 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:45.714789 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:45.714799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:45.714813 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:45.714848 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:45.717589 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.718326 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.718335 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.718338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.718494 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:45.718848 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:45.718852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:45.718854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:45.719262 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:45.719533 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:45.719721 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 05:06:45.719950 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":333,"threads_started":1}
I20260925 05:06:45.721871 9556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:45.803915 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9433
I20260925 05:06:45.821060 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9072
I20260925 05:06:45.822268 9134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:45.924221 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9072
2026-09-25T05:06:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3964 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:46 dist-test-slave-wpdm krb5kdc[9583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:46 dist-test-slave-wpdm krb5kdc[9583](info): set up 2 sockets
Sep 25 05:06:46 dist-test-slave-wpdm krb5kdc[9583](info): commencing operation
krb5kdc: starting...
W20260925 05:06:47.013633 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:06:47 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:06:47.192245 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35643
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:47.530495 9596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:47.530864 9596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:47.531051 9596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:47.538556 9596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:47.538718 9596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:47.538837 9596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:47.538949 9596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:47.539052 9596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:47.549098 9596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:35643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:47.550462 9596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:47.551779 9596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:47.563921 9604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:47.563887 9602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:47.564152 9601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:47.564703 9596 server_base.cc:1034] running on GCE node
I20260925 05:06:47.565474 9596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:47.566844 9596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:47.568064 9596 hybrid_clock.cc:648] HybridClock initialized: now 1790312807568020 us; error 59 us; skew 500 ppm
I20260925 05:06:47.568465 9596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:47 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:47.575897 9596 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:06:47.577404 9596 webserver.cc:460] Webserver started at http://127.8.215.62:46771/ using document root <none> and password file <none>
I20260925 05:06:47.578091 9596 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:47.578239 9596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:47.578608 9596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:47.581635 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "d169113e37204abbaf4455847195cc09"
format_stamp: "Formatted at 2026-09-25 05:06:47 on dist-test-slave-wpdm"
server_key: "c7be848564ef464b9b9e239032867632"
server_key_iv: "2fcf19c5f1fa2b098f3836778eaf2fbd"
server_key_version: "encryptionkey@0"
I20260925 05:06:47.582392 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "d169113e37204abbaf4455847195cc09"
format_stamp: "Formatted at 2026-09-25 05:06:47 on dist-test-slave-wpdm"
server_key: "c7be848564ef464b9b9e239032867632"
server_key_iv: "2fcf19c5f1fa2b098f3836778eaf2fbd"
server_key_version: "encryptionkey@0"
I20260925 05:06:47.587392 9596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:06:47.590837 9611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:47.592387 9596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:06:47.592574 9596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "d169113e37204abbaf4455847195cc09"
format_stamp: "Formatted at 2026-09-25 05:06:47 on dist-test-slave-wpdm"
server_key: "c7be848564ef464b9b9e239032867632"
server_key_iv: "2fcf19c5f1fa2b098f3836778eaf2fbd"
server_key_version: "encryptionkey@0"
I20260925 05:06:47.592823 9596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:47.606621 9596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:47.610745 9596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:47.611023 9596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:47.630395 9596 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:35643
I20260925 05:06:47.630414 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:35643 every 8 connection(s)
I20260925 05:06:47.631897 9596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:47.635762 9663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:47.639881 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9596
I20260925 05:06:47.640126 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:06:47.640913 9052 external_mini_cluster.cc:1428] Setting key ed94aeaf4ec56c61b1b409ba18ac5c18
I20260925 05:06:47.646798 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09: Bootstrap starting.
I20260925 05:06:47.650202 9663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:47.651213 9663 log.cc:824] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:47.653963 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09: No bootstrap required, opened a new log
Sep 25 05:06:47 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:47.661722 9663 raft_consensus.cc:348] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169113e37204abbaf4455847195cc09" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35643 } }
I20260925 05:06:47.662117 9663 raft_consensus.cc:374] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:47.662308 9663 raft_consensus.cc:729] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d169113e37204abbaf4455847195cc09, State: Initialized, Role: FOLLOWER
I20260925 05:06:47.662967 9663 consensus_queue.cc:260] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169113e37204abbaf4455847195cc09" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35643 } }
I20260925 05:06:47.663213 9663 raft_consensus.cc:388] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:47.663344 9663 raft_consensus.cc:482] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:47.663475 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:47.642948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.8.215.62:35643)
0925 05:06:47.643961 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:47.643973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:47.644018 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:06:47.645921 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:47.645944 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:47.646310 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:47.646679 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:47.648113 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:47.649497 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:47.650768 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:47.651501 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:47.651688 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:47.655959 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:47.655966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:47.655984 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:47.656022 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:47.659234 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:47.659900 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:47.659907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:47.659917 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:47.660098 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:47.660477 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:47.660485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:47.660487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:47.661082 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:47.661358 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:47.661844 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 05:06:47.662303 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":355,"threads_started":1}
I20260925 05:06:47.663529 9663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:47.665957 9663 raft_consensus.cc:504] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169113e37204abbaf4455847195cc09" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35643 } }
I20260925 05:06:47.666452 9663 leader_election.cc:302] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d169113e37204abbaf4455847195cc09; no voters:
I20260925 05:06:47.667088 9663 leader_election.cc:288] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:47.667213 9668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:47.667893 9668 raft_consensus.cc:686] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [term 1 LEADER]: Becoming Leader. State: Replica: d169113e37204abbaf4455847195cc09, State: Running, Role: LEADER
I20260925 05:06:47.668403 9668 consensus_queue.cc:237] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169113e37204abbaf4455847195cc09" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35643 } }
I20260925 05:06:47.669216 9663 sys_catalog.cc:564] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:47.671502 9669 sys_catalog.cc:455] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d169113e37204abbaf4455847195cc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169113e37204abbaf4455847195cc09" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35643 } } }
I20260925 05:06:47.671592 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d169113e37204abbaf4455847195cc09. Latest consensus state: current_term: 1 leader_uuid: "d169113e37204abbaf4455847195cc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169113e37204abbaf4455847195cc09" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35643 } } }
I20260925 05:06:47.671891 9669 sys_catalog.cc:458] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:47.671940 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:47.673861 9673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:47.680106 9673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:47.690572 9673 catalog_manager.cc:1347] Generated new cluster ID: a33e428544ae4945b85a6d52b08d7cfe
I20260925 05:06:47.690687 9673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:47.698750 9673 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:47.700500 9673 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:47.706553 9673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d169113e37204abbaf4455847195cc09: Generated new TSK 0
I20260925 05:06:47.707386 9673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:06:47.756827 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35643
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:48.078732 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:48.079003 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:48.079133 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:48.086596 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:48.086705 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:48.086875 9689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:06:48.096891 9689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:48.098228 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:48.099570 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:48.112757 9694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:48.112700 9697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:48.112782 9695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:48.113615 9689 server_base.cc:1034] running on GCE node
I20260925 05:06:48.114197 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:48.115124 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:48.116329 9689 hybrid_clock.cc:648] HybridClock initialized: now 1790312808116275 us; error 39 us; skew 500 ppm
I20260925 05:06:48.116662 9689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:48 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:48.119798 9689 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:06:48.121409 9689 webserver.cc:460] Webserver started at http://127.8.215.1:45121/ using document root <none> and password file <none>
I20260925 05:06:48.122117 9689 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:48.122260 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:48.122624 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:48.125638 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d"
format_stamp: "Formatted at 2026-09-25 05:06:48 on dist-test-slave-wpdm"
server_key: "eb6712749035620ada7e9f27104f7f3e"
server_key_iv: "557712fa110af8bf00081761e67ac623"
server_key_version: "encryptionkey@0"
I20260925 05:06:48.126425 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d"
format_stamp: "Formatted at 2026-09-25 05:06:48 on dist-test-slave-wpdm"
server_key: "eb6712749035620ada7e9f27104f7f3e"
server_key_iv: "557712fa110af8bf00081761e67ac623"
server_key_version: "encryptionkey@0"
I20260925 05:06:48.131891 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:06:48.135469 9704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:48.136999 9689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:06:48.137176 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d"
format_stamp: "Formatted at 2026-09-25 05:06:48 on dist-test-slave-wpdm"
server_key: "eb6712749035620ada7e9f27104f7f3e"
server_key_iv: "557712fa110af8bf00081761e67ac623"
server_key_version: "encryptionkey@0"
I20260925 05:06:48.137367 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:48.150671 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:48.155174 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:48.155474 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:48.156456 9689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:48.157847 9689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:48.157922 9689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:48.158035 9689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:48.158103 9689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:48.192385 9689 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:43389
I20260925 05:06:48.192423 9816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:43389 every 8 connection(s)
I20260925 05:06:48.193888 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:48.194514 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9689
I20260925 05:06:48.194764 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:48.195432 9052 external_mini_cluster.cc:1428] Setting key c14d385eba1f4820f054b50d3a655514
Sep 25 05:06:48 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
I20260925 05:06:48.214690 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:48.196695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:32823 (local address 127.8.215.62:35643)
0925 05:06:48.197209 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:48.197214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:48.198444 (+ 1230us) server_negotiation.cc:407] Connection header received
0925 05:06:48.200248 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:48.200256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:48.200346 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:48.200464 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:48.202065 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.202813 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.204370 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.204836 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.204968 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:48.208819 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:48.208825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:48.208828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:48.208859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:48.211431 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:48.212248 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:48.212260 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:48.212264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:48.212370 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:48.212854 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:48.212857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:48.212860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:48.213185 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:48.213336 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:48.214083 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 05:06:48.214293 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":258,"threads_started":1}
I20260925 05:06:48.215703 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:48.197437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35643 (local address 127.8.215.1:32823)
0925 05:06:48.198271 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:48.198323 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:48.199973 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:48.200692 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:48.200701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:48.201288 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:48.201884 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:48.201901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.202984 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.202988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:48.204216 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:48.204228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.205067 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.205075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:48.205323 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:48.206024 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:48.206046 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:48.208551 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:48.211581 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:48.211592 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:48.211610 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:48.212123 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:48.212480 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:48.212486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:48.212490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:48.212680 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:48.213310 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:48.213318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:48.213681 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 05:06:48.214013 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 05:06:48.214559 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":351,"threads_started":1}
I20260925 05:06:48.217578 9817 heartbeater.cc:347] Connected to a master server at 127.8.215.62:35643
I20260925 05:06:48.217883 9817 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:48.218623 9817 heartbeater.cc:511] Master 127.8.215.62:35643 requested a full tablet report, sending...
I20260925 05:06:48.220932 9628 ts_manager.cc:194] Registered new tserver with Master: c06d7f2a25ab42a1ae90c92edef9bb2d (127.8.215.1:43389)
I20260925 05:06:48.222360 9628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:32823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:06:48.236624 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35643
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:48.560797 9823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:48.561085 9823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:48.561187 9823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:48.568416 9823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:48.568521 9823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:48.568636 9823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:06:48.578433 9823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:48.579656 9823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:48.581023 9823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:48.594069 9831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:48.594133 9828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:48.594099 9829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:48.595521 9823 server_base.cc:1034] running on GCE node
I20260925 05:06:48.596132 9823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:48.597164 9823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:48.598364 9823 hybrid_clock.cc:648] HybridClock initialized: now 1790312808598312 us; error 63 us; skew 500 ppm
I20260925 05:06:48.598683 9823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:48 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:48.601751 9823 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:06:48.603124 9823 webserver.cc:460] Webserver started at http://127.8.215.2:36521/ using document root <none> and password file <none>
I20260925 05:06:48.603835 9823 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:48.603979 9823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:48.604295 9823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:48.607172 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "9903e46e66114f46a00d4a95882b1854"
format_stamp: "Formatted at 2026-09-25 05:06:48 on dist-test-slave-wpdm"
server_key: "8fa2c272f240771c6821d886fe73ef65"
server_key_iv: "1b650682b23b23e2c8e61c6d5bf35128"
server_key_version: "encryptionkey@0"
I20260925 05:06:48.607937 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "9903e46e66114f46a00d4a95882b1854"
format_stamp: "Formatted at 2026-09-25 05:06:48 on dist-test-slave-wpdm"
server_key: "8fa2c272f240771c6821d886fe73ef65"
server_key_iv: "1b650682b23b23e2c8e61c6d5bf35128"
server_key_version: "encryptionkey@0"
I20260925 05:06:48.613065 9823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 05:06:48.616464 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:48.617815 9823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:48.617986 9823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "9903e46e66114f46a00d4a95882b1854"
format_stamp: "Formatted at 2026-09-25 05:06:48 on dist-test-slave-wpdm"
server_key: "8fa2c272f240771c6821d886fe73ef65"
server_key_iv: "1b650682b23b23e2c8e61c6d5bf35128"
server_key_version: "encryptionkey@0"
I20260925 05:06:48.618178 9823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:48.635749 9823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:48.639742 9823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:48.640018 9823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:48.641321 9823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:48.642788 9823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:48.642889 9823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:48.643008 9823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:48.643090 9823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:48.675982 9823 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:33991
I20260925 05:06:48.676029 9950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:33991 every 8 connection(s)
I20260925 05:06:48.677299 9823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:48.684952 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9823
I20260925 05:06:48.685144 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:48.685560 9052 external_mini_cluster.cc:1428] Setting key a588e858d86a5d36420bf2acd459c54f
Sep 25 05:06:48 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:48.697774 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:48.679798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:38141 (local address 127.8.215.62:35643)
0925 05:06:48.679968 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:48.679973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:48.681246 (+ 1273us) server_negotiation.cc:407] Connection header received
0925 05:06:48.683089 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:48.683095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:48.683196 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:48.683343 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:48.684888 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.685703 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.687406 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.687964 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.688107 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:48.691379 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:48.691388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:48.691396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:48.691438 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:48.694160 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:48.695220 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:48.695224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:48.695226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:48.695350 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:48.695883 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:48.695887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:48.695889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:48.696181 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:48.696295 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:48.697285 (+ 990us) server_negotiation.cc:299] Negotiation successful
0925 05:06:48.697542 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:06:48.699481 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:48.680278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35643 (local address 127.8.215.2:38141)
0925 05:06:48.681080 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:48.681118 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:48.682832 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:48.683589 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:48.683598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:48.684096 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:48.684672 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:48.684689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.685863 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.685875 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:48.687262 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:48.687277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:48.688190 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:48.688195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:48.688380 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:48.689077 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:48.689102 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:48.691172 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:48.694314 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:48.694327 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:48.694347 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:48.695087 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:48.695459 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:48.695468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:48.695476 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:48.695766 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:48.696351 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:48.696364 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:48.696873 (+ 509us) client_negotiation.cc:769] Sending connection context
0925 05:06:48.697211 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 05:06:48.697853 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":264,"threads_started":1}
I20260925 05:06:48.701156 9951 heartbeater.cc:347] Connected to a master server at 127.8.215.62:35643
I20260925 05:06:48.701462 9951 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:48.702358 9951 heartbeater.cc:511] Master 127.8.215.62:35643 requested a full tablet report, sending...
I20260925 05:06:48.704176 9628 ts_manager.cc:194] Registered new tserver with Master: 9903e46e66114f46a00d4a95882b1854 (127.8.215.2:33991)
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
I20260925 05:06:48.705149 9628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:38141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:06:48.731417 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35643
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:49.072837 9956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:49.073150 9956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:49.073348 9956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:49.081017 9956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:49.081172 9956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:49.081393 9956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:06:49.091605 9956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:49.092993 9956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:49.094396 9956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:49.108129 9962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:49.108172 9961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:49.108201 9964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:49.109252 9956 server_base.cc:1034] running on GCE node
I20260925 05:06:49.109881 9956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:49.110915 9956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:49.112174 9956 hybrid_clock.cc:648] HybridClock initialized: now 1790312809112132 us; error 57 us; skew 500 ppm
I20260925 05:06:49.112514 9956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:49.115897 9956 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:06:49.117451 9956 webserver.cc:460] Webserver started at http://127.8.215.3:33231/ using document root <none> and password file <none>
I20260925 05:06:49.118141 9956 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:49.118237 9956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:49.118572 9956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:49.121658 9956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "4c2b6dba9b9a405299809a1e9d5ac709"
format_stamp: "Formatted at 2026-09-25 05:06:49 on dist-test-slave-wpdm"
server_key: "72b2aa01365cba0b44b469dd70cc6e9f"
server_key_iv: "87462823eabc04b1f3645e2006714c1a"
server_key_version: "encryptionkey@0"
I20260925 05:06:49.122583 9956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "4c2b6dba9b9a405299809a1e9d5ac709"
format_stamp: "Formatted at 2026-09-25 05:06:49 on dist-test-slave-wpdm"
server_key: "72b2aa01365cba0b44b469dd70cc6e9f"
server_key_iv: "87462823eabc04b1f3645e2006714c1a"
server_key_version: "encryptionkey@0"
I20260925 05:06:49.127712 9956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:06:49.131201 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:49.132480 9956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:06:49.132643 9956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "4c2b6dba9b9a405299809a1e9d5ac709"
format_stamp: "Formatted at 2026-09-25 05:06:49 on dist-test-slave-wpdm"
server_key: "72b2aa01365cba0b44b469dd70cc6e9f"
server_key_iv: "87462823eabc04b1f3645e2006714c1a"
server_key_version: "encryptionkey@0"
I20260925 05:06:49.132903 9956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:49.160590 9956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:49.165118 9956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:49.165468 9956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:49.166451 9956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:49.167773 9956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:49.167845 9956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:49.168011 9956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:49.168061 9956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:49.202330 9956 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39665
I20260925 05:06:49.202416 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39665 every 8 connection(s)
I20260925 05:06:49.203553 9956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:49.209613 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 9956
I20260925 05:06:49.209832 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:49.210353 9052 external_mini_cluster.cc:1428] Setting key 5898802b1c7690216e9e43f75ae644b5
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:49.224040 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.206209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:48353 (local address 127.8.215.62:35643)
0925 05:06:49.206655 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.206659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.207694 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 05:06:49.209609 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.209614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.209733 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.209864 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.211569 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.212585 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.214167 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.214571 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.214703 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.218169 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.218173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.218177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.218208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.220863 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.221666 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.221672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.221677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.221800 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.222227 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.222232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.222235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.222623 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.222772 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.223456 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.223664 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":245,"threads_started":1}
I20260925 05:06:49.225302 9817 heartbeater.cc:503] Master 127.8.215.62:35643 was elected leader, sending a full tablet report...
I20260925 05:06:49.225323 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.206694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35643 (local address 127.8.215.3:48353)
0925 05:06:49.207520 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.207567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:49.209332 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:49.210100 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:49.210108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:49.210705 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:49.211408 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.211428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.212780 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.212784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.214013 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.214029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.214814 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.214822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.215051 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.215706 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:49.215734 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:49.217952 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:49.221001 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:49.221010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:49.221027 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:49.221547 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:49.221907 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:49.221912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:49.221916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:49.222102 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:49.222755 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:49.222763 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:49.223097 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 05:06:49.223362 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 05:06:49.223899 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":291,"threads_started":1}
I20260925 05:06:49.226879 10084 heartbeater.cc:347] Connected to a master server at 127.8.215.62:35643
I20260925 05:06:49.227221 10084 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:49.227866 10084 heartbeater.cc:511] Master 127.8.215.62:35643 requested a full tablet report, sending...
I20260925 05:06:49.229148 9628 ts_manager.cc:194] Registered new tserver with Master: 4c2b6dba9b9a405299809a1e9d5ac709 (127.8.215.3:39665)
I20260925 05:06:49.229907 9628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:48353
I20260925 05:06:49.238468 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:06:49.256482 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.244768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.8.215.62:35643)
0925 05:06:49.245005 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.245013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.245529 (+ 516us) server_negotiation.cc:407] Connection header received
0925 05:06:49.245666 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.245669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.245731 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.245810 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.247267 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.247971 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.249248 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.249651 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.249741 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.251966 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.251970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.251973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.251999 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.254282 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.254855 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.254859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.254861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.254980 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.255290 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.255293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.255295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.255589 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.255765 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.256066 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.256222 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:06:49.262874 9628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:49.270411 9628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:49.305158 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.285382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:33991 (local address 127.0.0.1:44950)
0925 05:06:49.296343 (+ 10961us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.296361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:49.296455 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:49.297197 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:49.297202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:49.297229 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:49.297647 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.297660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.299553 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.299560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.302260 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.302273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.304205 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.304212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.304402 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.304434 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:06:49.304502 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 05:06:49.304665 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10826,"mutex_wait_us":94,"thread_start_us":10703,"threads_started":1}
I20260925 05:06:49.305184 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.282570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.8.215.1:43389)
0925 05:06:49.294932 (+ 12362us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.294945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.295561 (+ 616us) server_negotiation.cc:407] Connection header received
0925 05:06:49.295670 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.295678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.295970 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.296269 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:49.297495 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.299093 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.301642 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.303676 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.303844 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.304118 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.304632 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.304851 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12132,"thread_start_us":12377,"threads_started":1}
I20260925 05:06:49.305426 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.282988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.8.215.3:39665)
0925 05:06:49.297205 (+ 14217us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.297213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.297242 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:06:49.297340 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.297347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.297640 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.297824 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:49.298400 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.299940 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.302770 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.304184 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.304314 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.304522 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.304821 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.304983 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14032,"thread_start_us":13870,"threads_started":1}
I20260925 05:06:49.306041 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.282620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39665 (local address 127.0.0.1:45438)
0925 05:06:49.295680 (+ 13060us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.295701 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:49.295841 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:49.297835 (+ 1994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:49.297839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:49.297863 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:49.298210 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.298222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.300101 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.300106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.302588 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.302613 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.304405 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.304411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.304577 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.304639 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 05:06:49.304749 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:06:49.304879 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12906,"mutex_wait_us":30,"thread_start_us":11854,"threads_started":1}
I20260925 05:06:49.306154 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.285137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.8.215.2:33991)
0925 05:06:49.296540 (+ 11403us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.296549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.296572 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:06:49.296667 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.296674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.296933 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.297188 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:49.298063 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.299426 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.302450 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.303963 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.305427 (+ 1464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.305623 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.305699 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.305860 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11241,"thread_start_us":11135,"threads_started":1}
I20260925 05:06:49.306159 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.282397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:43389 (local address 127.0.0.1:47472)
0925 05:06:49.294896 (+ 12499us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.294946 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:49.295164 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:49.296544 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:49.296551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:49.296654 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:49.297035 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.297051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.299260 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.299264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.301489 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.301500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.303957 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.303964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.304173 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.304220 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:06:49.304390 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 05:06:49.305755 (+ 1365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12226,"mutex_wait_us":1162,"thread_start_us":12949,"threads_started":1}
I20260925 05:06:49.309831 9752 tablet_service.cc:1467] Processing CreateTablet for tablet fb0cefb7f87543a5b7c5c343a280e0fc (DEFAULT_TABLE table=test-table [id=e352dfc79b204b84914e2faecb334370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:49.309849 10019 tablet_service.cc:1467] Processing CreateTablet for tablet fb0cefb7f87543a5b7c5c343a280e0fc (DEFAULT_TABLE table=test-table [id=e352dfc79b204b84914e2faecb334370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:49.309849 9886 tablet_service.cc:1467] Processing CreateTablet for tablet fb0cefb7f87543a5b7c5c343a280e0fc (DEFAULT_TABLE table=test-table [id=e352dfc79b204b84914e2faecb334370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:49.311100 9752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0cefb7f87543a5b7c5c343a280e0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:49.311156 10019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0cefb7f87543a5b7c5c343a280e0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:49.311162 9886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0cefb7f87543a5b7c5c343a280e0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:49.323439 10103 tablet_bootstrap.cc:492] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: Bootstrap starting.
I20260925 05:06:49.325960 10105 tablet_bootstrap.cc:492] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: Bootstrap starting.
I20260925 05:06:49.327680 10104 tablet_bootstrap.cc:492] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: Bootstrap starting.
I20260925 05:06:49.328498 10103 tablet_bootstrap.cc:654] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:49.330016 10103 log.cc:824] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:49.331125 10104 tablet_bootstrap.cc:654] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:49.331162 10105 tablet_bootstrap.cc:654] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:49.332167 10105 log.cc:824] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:49.332463 10104 log.cc:824] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:49.333431 10103 tablet_bootstrap.cc:492] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: No bootstrap required, opened a new log
I20260925 05:06:49.333722 10103 ts_tablet_manager.cc:1397] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 05:06:49.335137 10105 tablet_bootstrap.cc:492] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: No bootstrap required, opened a new log
I20260925 05:06:49.335404 10105 ts_tablet_manager.cc:1397] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260925 05:06:49.336309 10104 tablet_bootstrap.cc:492] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: No bootstrap required, opened a new log
I20260925 05:06:49.336614 10104 ts_tablet_manager.cc:1397] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:06:49.343509 10105 raft_consensus.cc:348] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.343931 10105 raft_consensus.cc:374] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:49.344023 10105 raft_consensus.cc:729] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2b6dba9b9a405299809a1e9d5ac709, State: Initialized, Role: FOLLOWER
I20260925 05:06:49.344687 10105 consensus_queue.cc:260] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.344918 10103 raft_consensus.cc:348] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.345357 10103 raft_consensus.cc:374] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:49.345455 10103 raft_consensus.cc:729] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06d7f2a25ab42a1ae90c92edef9bb2d, State: Initialized, Role: FOLLOWER
I20260925 05:06:49.346069 10084 heartbeater.cc:503] Master 127.8.215.62:35643 was elected leader, sending a full tablet report...
I20260925 05:06:49.346848 10105 ts_tablet_manager.cc:1428] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 05:06:49.347335 10104 raft_consensus.cc:348] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.347779 10104 raft_consensus.cc:374] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:49.347894 10104 raft_consensus.cc:729] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903e46e66114f46a00d4a95882b1854, State: Initialized, Role: FOLLOWER
I20260925 05:06:49.347841 10103 consensus_queue.cc:260] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.348511 10104 consensus_queue.cc:260] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.349807 10103 ts_tablet_manager.cc:1428] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260925 05:06:49.350741 9951 heartbeater.cc:503] Master 127.8.215.62:35643 was elected leader, sending a full tablet report...
I20260925 05:06:49.351395 10104 ts_tablet_manager.cc:1428] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260925 05:06:49.429248 9952 tablet.cc:2369] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:06:49.446481 9818 tablet.cc:2369] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:06:49.455389 10085 tablet.cc:2369] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:49.474046 10111 raft_consensus.cc:482] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:49.474375 10111 raft_consensus.cc:504] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.476119 10111 leader_election.cc:288] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c06d7f2a25ab42a1ae90c92edef9bb2d (127.8.215.1:43389), 4c2b6dba9b9a405299809a1e9d5ac709 (127.8.215.3:39665)
I20260925 05:06:49.482952 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.476651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:49675 (local address 127.8.215.1:43389)
0925 05:06:49.476902 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.476910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.477557 (+ 647us) server_negotiation.cc:407] Connection header received
0925 05:06:49.477786 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.477790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.477846 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.477944 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:49.478488 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.479270 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.481086 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.482038 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.482196 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.482301 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.482573 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.482695 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 05:06:49.483141 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.476509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:43389 (local address 127.8.215.2:49675)
0925 05:06:49.477430 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.477448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:49.477627 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:49.477958 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:49.477962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:49.477984 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:49.478342 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.478352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.479395 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.479403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.480924 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.480935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.482261 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.482268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.482410 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.482437 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:06:49.482550 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:06:49.482685 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":546,"threads_started":1}
I20260925 05:06:49.483181 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.476798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:56655 (local address 127.8.215.3:39665)
0925 05:06:49.477113 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.477119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.477750 (+ 631us) server_negotiation.cc:407] Connection header received
0925 05:06:49.477948 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.477951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.478006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.478109 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:49.478714 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.479485 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.481489 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.482265 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.482414 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.482515 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.482804 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.482921 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260925 05:06:49.483592 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.476634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39665 (local address 127.8.215.2:56655)
0925 05:06:49.477643 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.477660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:49.477816 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:49.478119 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:49.478125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:49.478152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:49.478563 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.478572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.479635 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.479640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.481339 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:49.481356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.482477 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.482482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:49.482656 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.482678 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:06:49.482780 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:06:49.482903 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":677,"threads_started":1}
I20260925 05:06:49.484308 9772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0cefb7f87543a5b7c5c343a280e0fc" candidate_uuid: "9903e46e66114f46a00d4a95882b1854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" is_pre_election: true
I20260925 05:06:49.484568 10039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0cefb7f87543a5b7c5c343a280e0fc" candidate_uuid: "9903e46e66114f46a00d4a95882b1854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" is_pre_election: true
I20260925 05:06:49.484974 9772 raft_consensus.cc:2454] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903e46e66114f46a00d4a95882b1854 in term 0.
I20260925 05:06:49.485146 10039 raft_consensus.cc:2454] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903e46e66114f46a00d4a95882b1854 in term 0.
I20260925 05:06:49.485644 9839 leader_election.cc:302] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903e46e66114f46a00d4a95882b1854, c06d7f2a25ab42a1ae90c92edef9bb2d; no voters:
I20260925 05:06:49.486042 10111 raft_consensus.cc:2781] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:49.486189 10111 raft_consensus.cc:482] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:49.486307 10111 raft_consensus.cc:3037] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:49.488399 10111 raft_consensus.cc:504] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.489091 10111 leader_election.cc:288] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [CANDIDATE]: Term 1 election: Requested vote from peers c06d7f2a25ab42a1ae90c92edef9bb2d (127.8.215.1:43389), 4c2b6dba9b9a405299809a1e9d5ac709 (127.8.215.3:39665)
I20260925 05:06:49.489487 9772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0cefb7f87543a5b7c5c343a280e0fc" candidate_uuid: "9903e46e66114f46a00d4a95882b1854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d"
I20260925 05:06:49.489621 10039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0cefb7f87543a5b7c5c343a280e0fc" candidate_uuid: "9903e46e66114f46a00d4a95882b1854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2b6dba9b9a405299809a1e9d5ac709"
I20260925 05:06:49.489643 9772 raft_consensus.cc:3037] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:49.489838 10039 raft_consensus.cc:3037] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:49.491847 9772 raft_consensus.cc:2454] T fb0cefb7f87543a5b7c5c343a280e0fc P c06d7f2a25ab42a1ae90c92edef9bb2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903e46e66114f46a00d4a95882b1854 in term 1.
I20260925 05:06:49.492050 10039 raft_consensus.cc:2454] T fb0cefb7f87543a5b7c5c343a280e0fc P 4c2b6dba9b9a405299809a1e9d5ac709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903e46e66114f46a00d4a95882b1854 in term 1.
I20260925 05:06:49.492210 9839 leader_election.cc:302] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903e46e66114f46a00d4a95882b1854, c06d7f2a25ab42a1ae90c92edef9bb2d; no voters:
I20260925 05:06:49.492484 10111 raft_consensus.cc:2781] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:49.493017 10111 raft_consensus.cc:686] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [term 1 LEADER]: Becoming Leader. State: Replica: 9903e46e66114f46a00d4a95882b1854, State: Running, Role: LEADER
I20260925 05:06:49.493472 10111 consensus_queue.cc:237] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } }
I20260925 05:06:49.498155 9626 catalog_manager.cc:5495] T fb0cefb7f87543a5b7c5c343a280e0fc P 9903e46e66114f46a00d4a95882b1854 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903e46e66114f46a00d4a95882b1854 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "9903e46e66114f46a00d4a95882b1854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06d7f2a25ab42a1ae90c92edef9bb2d" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c2b6dba9b9a405299809a1e9d5ac709" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9903e46e66114f46a00d4a95882b1854" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33991 } health_report { overall_health: HEALTHY } } }
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:06:49.551136 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.535486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.8.215.1:43389)
0925 05:06:49.535678 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.535683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.536384 (+ 701us) server_negotiation.cc:407] Connection header received
0925 05:06:49.536504 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.536508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.536591 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.536684 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.538344 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.539105 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.540491 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.541143 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.541274 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.545362 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.545376 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.545395 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.545436 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.548747 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.549371 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.549379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.549381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.549521 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.549829 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.549832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.549834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.550249 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.550489 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.550679 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.550842 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:06:49.597266 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.583300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.8.215.1:43389)
0925 05:06:49.583547 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.583557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.583940 (+ 383us) server_negotiation.cc:407] Connection header received
0925 05:06:49.584108 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.584113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.584179 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.584264 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.585741 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.586455 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.587765 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.588160 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.588251 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.592058 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.592063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.592066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.592095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.594792 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.595396 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.595399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.595402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.595524 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.595924 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.595928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.595930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.596189 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.596389 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.596612 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.596873 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:06:49.644032 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.627748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.8.215.1:43389)
0925 05:06:49.628003 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.628008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.628159 (+ 151us) server_negotiation.cc:407] Connection header received
0925 05:06:49.628304 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.628308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.628371 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.628452 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.629802 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.630922 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.632175 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.632843 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.632957 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.637697 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.637704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.637709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.637748 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.641212 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.642011 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.642016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.642020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.642157 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.642523 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.642527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.642530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.642873 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.643125 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.643466 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.643691 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260925 05:06:49.646081 9732 tablet_service.cc:2908] Rejecting scan request for tablet fb0cefb7f87543a5b7c5c343a280e0fc: Uninitialized: clean time has not yet been initialized
I20260925 05:06:49.649072 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9689
I20260925 05:06:49.661024 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.650041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.8.215.1:43389)
0925 05:06:49.650197 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.650201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.650643 (+ 442us) server_negotiation.cc:407] Connection header received
0925 05:06:49.650758 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.650763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.650826 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.650900 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.651957 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.652596 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.653797 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.654192 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.654286 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.656329 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.656333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.656336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.656363 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.658778 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.659383 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.659386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.659388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.659493 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.659858 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.659861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.659863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.660126 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.660327 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.660581 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.660793 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:06:49.661768 9812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:49.767030 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9689
I20260925 05:06:49.787352 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9823
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:06:49.804684 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.788136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.8.215.2:33991)
0925 05:06:49.788355 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.788363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.788387 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:06:49.788454 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.788460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.788542 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.788665 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.789880 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.791160 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.792452 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.793136 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.793224 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.797733 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.797754 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.797777 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.797839 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.801788 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.802434 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.802445 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.802450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.802660 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.802998 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.803004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.803007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.803500 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.803849 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.804008 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.804252 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:06:49.805603 9946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:49.895315 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.894871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:43389 (local address 127.8.215.2:51919)
0925 05:06:49.895054 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:49.895185 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 05:06:49.895293 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9823
I20260925 05:06:49.912920 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9956
Sep 25 05:06:49 dist-test-slave-wpdm krb5kdc[9583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:06:49.927884 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:49.913799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.8.215.3:39665)
0925 05:06:49.913983 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:49.913989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:49.914082 (+ 93us) server_negotiation.cc:407] Connection header received
0925 05:06:49.914315 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:49.914319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:49.914381 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:49.914485 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:49.915653 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.916350 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.917703 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:49.918090 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:49.918221 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:49.922482 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:49.922495 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:49.922512 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:49.922549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:49.925165 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.925826 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.925835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.925837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.925973 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:49.926366 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:49.926373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:49.926376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:49.926726 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:49.927009 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:49.927314 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 05:06:49.927540 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:06:49.928516 10079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:50.016068 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9956
I20260925 05:06:50.032330 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 9596
I20260925 05:06:50.033449 9658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:50.139442 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 9596
2026-09-25T05:06:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4213 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:50 dist-test-slave-wpdm krb5kdc[10133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:50 dist-test-slave-wpdm krb5kdc[10133](info): set up 2 sockets
Sep 25 05:06:50 dist-test-slave-wpdm krb5kdc[10133](info): commencing operation
krb5kdc: starting...
W20260925 05:06:51.225345 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:06:51 dist-test-slave-wpdm krb5kdc[10133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:06:51.412595 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:33227
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:51.746119 10146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:51.746541 10146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:51.746734 10146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:51.754298 10146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:51.754468 10146 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:06:51.754576 10146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:51.754714 10146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:51.754837 10146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:51.754956 10146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:51.765204 10146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:33227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:51.766626 10146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:51.768086 10146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:51.781244 10151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:51.781206 10152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:51.781158 10154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:51.782245 10146 server_base.cc:1034] running on GCE node
I20260925 05:06:51.783140 10146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:51.784582 10146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:51.785938 10146 hybrid_clock.cc:648] HybridClock initialized: now 1790312811785861 us; error 70 us; skew 500 ppm
I20260925 05:06:51.786342 10146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:51 dist-test-slave-wpdm krb5kdc[10133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:51.789505 10146 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:06:51.790939 10146 webserver.cc:460] Webserver started at http://127.8.215.62:38291/ using document root <none> and password file <none>
I20260925 05:06:51.791637 10146 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:51.791770 10146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:51.792094 10146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:51.795090 10146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "c198cd0d50174f15aa889f5f28e32fd8"
format_stamp: "Formatted at 2026-09-25 05:06:51 on dist-test-slave-wpdm"
server_key: "2d416326e015a1997b9d2e98ff4da60e"
server_key_iv: "f5bf78063ab9dd0e97fd2641f9e46ed1"
server_key_version: "encryptionkey@0"
I20260925 05:06:51.795869 10146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "c198cd0d50174f15aa889f5f28e32fd8"
format_stamp: "Formatted at 2026-09-25 05:06:51 on dist-test-slave-wpdm"
server_key: "2d416326e015a1997b9d2e98ff4da60e"
server_key_iv: "f5bf78063ab9dd0e97fd2641f9e46ed1"
server_key_version: "encryptionkey@0"
I20260925 05:06:51.800887 10146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 05:06:51.804154 10161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:51.805490 10146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:51.805665 10146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "c198cd0d50174f15aa889f5f28e32fd8"
format_stamp: "Formatted at 2026-09-25 05:06:51 on dist-test-slave-wpdm"
server_key: "2d416326e015a1997b9d2e98ff4da60e"
server_key_iv: "f5bf78063ab9dd0e97fd2641f9e46ed1"
server_key_version: "encryptionkey@0"
I20260925 05:06:51.805915 10146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:51.835598 10146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:51.840085 10146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:51.840366 10146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:51.862236 10146 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:33227
I20260925 05:06:51.862243 10222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:33227 every 8 connection(s)
I20260925 05:06:51.863623 10146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:51.867491 10223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:51.870507 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 10146
I20260925 05:06:51.870728 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:06:51.871531 9052 external_mini_cluster.cc:1428] Setting key 076b490cca3f8bb351b704b2d5678c24
I20260925 05:06:51.879671 10223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: Bootstrap starting.
I20260925 05:06:51.882747 10223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:51.884132 10223 log.cc:824] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:06:51 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:51.887059 10223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: No bootstrap required, opened a new log
I20260925 05:06:51.894136 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:51.873808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.8.215.62:33227)
0925 05:06:51.874542 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:51.874554 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:51.874595 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:06:51.876312 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:51.876336 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:51.876804 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:51.877250 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:51.878658 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:51.880011 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:51.881292 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:51.882023 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:51.882207 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:51.886813 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:51.886820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:51.886835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:51.886871 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:51.890025 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:51.890754 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:51.890761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:51.890768 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:51.890938 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:51.891337 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:51.891341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:51.891343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:51.891877 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:51.892189 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:51.892637 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 05:06:51.893100 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":335,"threads_started":1}
I20260925 05:06:51.895545 10223 raft_consensus.cc:348] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198cd0d50174f15aa889f5f28e32fd8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33227 } }
I20260925 05:06:51.896008 10223 raft_consensus.cc:374] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:51.896163 10223 raft_consensus.cc:729] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c198cd0d50174f15aa889f5f28e32fd8, State: Initialized, Role: FOLLOWER
I20260925 05:06:51.896903 10223 consensus_queue.cc:260] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198cd0d50174f15aa889f5f28e32fd8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33227 } }
I20260925 05:06:51.897200 10223 raft_consensus.cc:388] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:51.897359 10223 raft_consensus.cc:482] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:51.897562 10223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:51.899802 10223 raft_consensus.cc:504] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198cd0d50174f15aa889f5f28e32fd8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33227 } }
I20260925 05:06:51.900276 10223 leader_election.cc:302] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c198cd0d50174f15aa889f5f28e32fd8; no voters:
I20260925 05:06:51.900918 10223 leader_election.cc:288] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:51.901072 10228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:51.901839 10228 raft_consensus.cc:686] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c198cd0d50174f15aa889f5f28e32fd8, State: Running, Role: LEADER
I20260925 05:06:51.902470 10228 consensus_queue.cc:237] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198cd0d50174f15aa889f5f28e32fd8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33227 } }
I20260925 05:06:51.903476 10223 sys_catalog.cc:564] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:51.906064 10229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c198cd0d50174f15aa889f5f28e32fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198cd0d50174f15aa889f5f28e32fd8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33227 } } }
I20260925 05:06:51.906031 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c198cd0d50174f15aa889f5f28e32fd8. Latest consensus state: current_term: 1 leader_uuid: "c198cd0d50174f15aa889f5f28e32fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198cd0d50174f15aa889f5f28e32fd8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33227 } } }
I20260925 05:06:51.906488 10229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:51.906529 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:51.907727 10235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:51.914235 10235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:51.926980 10235 catalog_manager.cc:1347] Generated new cluster ID: 03e2f2692d6b4bdd8b4db03b823b4442
I20260925 05:06:51.927114 10235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:51.939483 10235 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:51.941498 10235 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:51.948766 10235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: Generated new TSK 0
I20260925 05:06:51.949637 10235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:06:52.006688 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33227
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:52.347613 10249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:52.347908 10249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:52.348121 10249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:52.355491 10249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:52.355602 10249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:52.355757 10249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:52.355892 10249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:06:52.365780 10249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:52.367074 10249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:52.368413 10249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:52.381601 10254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:52.381644 10257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:52.381580 10255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:52.382553 10249 server_base.cc:1034] running on GCE node
I20260925 05:06:52.383148 10249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:52.384061 10249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:52.385304 10249 hybrid_clock.cc:648] HybridClock initialized: now 1790312812385239 us; error 42 us; skew 500 ppm
I20260925 05:06:52.385704 10249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:52 dist-test-slave-wpdm krb5kdc[10133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:52.388798 10249 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:06:52.390141 10249 webserver.cc:460] Webserver started at http://127.8.215.1:35399/ using document root <none> and password file <none>
I20260925 05:06:52.390816 10249 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:52.390944 10249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:52.391266 10249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:52.394156 10249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "a30fc178d88e4986a54cbd7104a14abe"
format_stamp: "Formatted at 2026-09-25 05:06:52 on dist-test-slave-wpdm"
server_key: "e8cea1b611c10e8285cd6c733f2fc18e"
server_key_iv: "87ccc8c931ae01b54ec2a0f7c7b6e1ed"
server_key_version: "encryptionkey@0"
I20260925 05:06:52.394927 10249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "a30fc178d88e4986a54cbd7104a14abe"
format_stamp: "Formatted at 2026-09-25 05:06:52 on dist-test-slave-wpdm"
server_key: "e8cea1b611c10e8285cd6c733f2fc18e"
server_key_iv: "87ccc8c931ae01b54ec2a0f7c7b6e1ed"
server_key_version: "encryptionkey@0"
I20260925 05:06:52.399757 10249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 05:06:52.403062 10264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:52.404484 10249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:06:52.404659 10249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "a30fc178d88e4986a54cbd7104a14abe"
format_stamp: "Formatted at 2026-09-25 05:06:52 on dist-test-slave-wpdm"
server_key: "e8cea1b611c10e8285cd6c733f2fc18e"
server_key_iv: "87ccc8c931ae01b54ec2a0f7c7b6e1ed"
server_key_version: "encryptionkey@0"
I20260925 05:06:52.404958 10249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:52.431275 10249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:52.435473 10249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:52.435771 10249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:52.438498 10249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:52.438607 10249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:52.438771 10249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:52.438836 10249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:06:52 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:52.462604 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.438956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:52699 (local address 127.8.215.62:33227)
0925 05:06:52.439688 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:52.439692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:52.440622 (+ 930us) server_negotiation.cc:407] Connection header received
0925 05:06:52.442996 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:52.443001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:52.443091 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:52.443230 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:52.446034 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.447113 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.449401 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.449964 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.450095 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.454933 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:52.454942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:52.454949 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:52.454994 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:52.458154 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:52.459281 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:52.459285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:52.459287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:52.459405 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:52.460023 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:52.460028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:52.460032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:52.460418 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:52.460558 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:52.461731 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0925 05:06:52.462040 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":538,"threads_started":1}
I20260925 05:06:52.463315 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.439644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.8.215.1:52699)
0925 05:06:52.440464 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:52.440499 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:52.442719 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:52.443735 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:52.443752 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:52.444746 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:52.445792 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.445821 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.447310 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.447314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.449248 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.449260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.450231 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.450242 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.450597 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.451460 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:52.451482 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:52.454605 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:52.458324 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:52.458338 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:52.458365 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:52.459136 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:52.459526 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:52.459531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:52.459535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:52.459860 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:52.460627 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:52.460635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:52.461234 (+ 599us) client_negotiation.cc:769] Sending connection context
0925 05:06:52.461603 (+ 369us) client_negotiation.cc:240] Negotiation successful
0925 05:06:52.462082 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":369,"threads_started":1}
I20260925 05:06:52.486186 10249 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:36539
I20260925 05:06:52.486207 10384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:36539 every 8 connection(s)
I20260925 05:06:52.487859 10249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:52.489506 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.471289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.8.215.1:32859)
0925 05:06:52.472228 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:52.472256 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:52.472441 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:52.472883 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:52.472893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:52.473847 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:52.474388 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.474404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.475982 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.475992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.477779 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.477792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.479080 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.479086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.479375 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.480169 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:52.480199 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:52.481880 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:52.486207 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:52.486215 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:52.486222 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:52.486742 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:52.487208 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:52.487212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:52.487214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:52.487342 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:52.488072 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:52.488077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:52.488299 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 05:06:52.488468 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 05:06:52.488919 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":545,"threads_started":1}
I20260925 05:06:52.490145 10385 heartbeater.cc:347] Connected to a master server at 127.8.215.62:33227
I20260925 05:06:52.490388 10385 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:52.490412 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.471289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:32859 (local address 127.8.215.62:33227)
0925 05:06:52.471512 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:52.471521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:52.472366 (+ 845us) server_negotiation.cc:407] Connection header received
0925 05:06:52.472615 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:52.472622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:52.472707 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:52.472851 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:52.474552 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.475828 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.477929 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.478628 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.478783 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.482163 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:52.482173 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:52.482180 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:52.482231 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:52.486029 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:52.486907 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:52.486912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:52.486915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:52.487071 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:52.487548 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:52.487552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:52.487555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:52.487923 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:52.489781 (+ 1858us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:52.489925 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 05:06:52.490139 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:52.491039 10385 heartbeater.cc:511] Master 127.8.215.62:33227 requested a full tablet report, sending...
I20260925 05:06:52.493121 10178 ts_manager.cc:194] Registered new tserver with Master: a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539)
I20260925 05:06:52.494670 10178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:52699
I20260925 05:06:52.497951 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 10249
I20260925 05:06:52.498196 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:52.498816 9052 external_mini_cluster.cc:1428] Setting key c2e48b9c3beb24a8afe746591505eba4
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:06:52.540110 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33227
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:52.866236 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:52.866546 10389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:52.866717 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:52.874114 10389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:52.874264 10389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:52.874426 10389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:52.874590 10389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:06:52.884925 10389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:52.886267 10389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:52.887576 10389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:52.900652 10395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:52.900738 10394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:52.900696 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:52.902019 10389 server_base.cc:1034] running on GCE node
I20260925 05:06:52.902616 10389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:52.903553 10389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:52.904850 10389 hybrid_clock.cc:648] HybridClock initialized: now 1790312812904723 us; error 62 us; skew 500 ppm
I20260925 05:06:52.905174 10389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:52 dist-test-slave-wpdm krb5kdc[10133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:52.913193 10389 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:06:52.914736 10389 webserver.cc:460] Webserver started at http://127.8.215.2:41471/ using document root <none> and password file <none>
I20260925 05:06:52.915490 10389 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:52.915647 10389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:52.915977 10389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:52.918891 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "6b52894ac33f4c0d943958ee5374551b"
format_stamp: "Formatted at 2026-09-25 05:06:52 on dist-test-slave-wpdm"
server_key: "eec7c403d7dd61e2c25c2d028d4ee6f2"
server_key_iv: "bebef1810bc801891f0defa44f8dbd7c"
server_key_version: "encryptionkey@0"
I20260925 05:06:52.919677 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "6b52894ac33f4c0d943958ee5374551b"
format_stamp: "Formatted at 2026-09-25 05:06:52 on dist-test-slave-wpdm"
server_key: "eec7c403d7dd61e2c25c2d028d4ee6f2"
server_key_iv: "bebef1810bc801891f0defa44f8dbd7c"
server_key_version: "encryptionkey@0"
I20260925 05:06:52.924845 10389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:06:52.928293 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:52.929790 10389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:06:52.929989 10389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "6b52894ac33f4c0d943958ee5374551b"
format_stamp: "Formatted at 2026-09-25 05:06:52 on dist-test-slave-wpdm"
server_key: "eec7c403d7dd61e2c25c2d028d4ee6f2"
server_key_iv: "bebef1810bc801891f0defa44f8dbd7c"
server_key_version: "encryptionkey@0"
I20260925 05:06:52.930228 10389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:52.949157 10389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:52.953263 10389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:52.953544 10389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:52.956421 10389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:52.956532 10389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:52.956698 10389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:52.956776 10389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 05:06:52 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:52.980991 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.956884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:57373 (local address 127.8.215.62:33227)
0925 05:06:52.957080 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:52.957088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:52.958584 (+ 1496us) server_negotiation.cc:407] Connection header received
0925 05:06:52.960788 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:52.960794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:52.960893 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:52.961003 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:52.963480 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.964417 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.967261 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.967826 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.967976 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.973211 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:52.973219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:52.973227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:52.973277 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:52.977222 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:52.978036 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:52.978045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:52.978051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:52.978237 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:52.978708 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:52.978716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:52.978722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:52.979232 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:52.979417 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:52.980117 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 05:06:52.980417 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:06:52.982082 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.957575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.8.215.2:57373)
0925 05:06:52.958410 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:52.958457 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:52.960424 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:52.961324 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:52.961340 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:52.962240 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:52.963198 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.963231 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.964582 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.964588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.967111 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.967124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.968079 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.968087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.968364 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.969226 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:52.969265 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:52.972906 (+ 3641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:52.977398 (+ 4492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:52.977406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:52.977421 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:52.977879 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:52.978365 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:52.978368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:52.978371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:52.978580 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:52.979369 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:52.979376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:52.979727 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 05:06:52.979959 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 05:06:52.980360 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":398,"threads_started":1}
I20260925 05:06:53.006482 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.989631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:55585 (local address 127.8.215.62:33227)
0925 05:06:52.989853 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:52.989862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:52.990747 (+ 885us) server_negotiation.cc:407] Connection header received
0925 05:06:52.990986 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:52.990992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:52.991071 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:52.991188 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:52.992208 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.993110 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.996205 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.996623 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.996800 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.000027 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:53.000042 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:53.000049 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:53.000093 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:53.003408 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.004424 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.004429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.004431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.004558 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.005051 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.005056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.005059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.005376 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:53.005552 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.005919 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.006138 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:06:53.006642 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:52.989717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.8.215.2:55585)
0925 05:06:52.990634 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:52.990651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:52.990825 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:52.991212 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:52.991217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:52.991735 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:52.992055 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.992064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.993345 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.993352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.996000 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:52.996025 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:52.996892 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:52.996903 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:52.997216 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:52.998047 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:52.998073 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:52.999796 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:53.003561 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:53.003566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:53.003571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:53.004288 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:53.004679 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:53.004684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:53.004688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:53.004903 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:53.005495 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:53.005499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:53.005687 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.005832 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.006188 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":576,"threads_started":1}
I20260925 05:06:53.018579 10389 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:40285
I20260925 05:06:53.018644 10523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:40285 every 8 connection(s)
I20260925 05:06:53.019928 10389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:53.022178 10524 heartbeater.cc:347] Connected to a master server at 127.8.215.62:33227
I20260925 05:06:53.022414 10524 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:53.023073 10524 heartbeater.cc:511] Master 127.8.215.62:33227 requested a full tablet report, sending...
I20260925 05:06:53.024389 10178 ts_manager.cc:194] Registered new tserver with Master: 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285)
I20260925 05:06:53.025357 10178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:57373
I20260925 05:06:53.028956 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 10389
I20260925 05:06:53.029186 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:53.029615 9052 external_mini_cluster.cc:1428] Setting key c4edee29fdf74bc8e8760728a764ccd8
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:06:53.070207 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33227
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:53.403765 10528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:53.404119 10528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:53.404313 10528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:53.412164 10528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:53.412282 10528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:53.412447 10528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:53.412536 10528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:06:53.422665 10528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:53.424002 10528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:53.425529 10528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:53.439714 10533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:53.439805 10534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:53.439687 10536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:53.440958 10528 server_base.cc:1034] running on GCE node
I20260925 05:06:53.441639 10528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:53.442629 10528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:53.443867 10528 hybrid_clock.cc:648] HybridClock initialized: now 1790312813443784 us; error 64 us; skew 500 ppm
I20260925 05:06:53.444168 10528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:53 dist-test-slave-wpdm krb5kdc[10133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:53.447479 10528 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:06:53.449136 10528 webserver.cc:460] Webserver started at http://127.8.215.3:32967/ using document root <none> and password file <none>
I20260925 05:06:53.449855 10528 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:53.449970 10528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:53.450316 10528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:53.453296 10528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "c5671cb02dfc4145bfc344b88006fa9e"
format_stamp: "Formatted at 2026-09-25 05:06:53 on dist-test-slave-wpdm"
server_key: "9435227bc46d736b5958352e1d8116b2"
server_key_iv: "02ee66169f8c4154b822a70dc49a1cfd"
server_key_version: "encryptionkey@0"
I20260925 05:06:53.454106 10528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "c5671cb02dfc4145bfc344b88006fa9e"
format_stamp: "Formatted at 2026-09-25 05:06:53 on dist-test-slave-wpdm"
server_key: "9435227bc46d736b5958352e1d8116b2"
server_key_iv: "02ee66169f8c4154b822a70dc49a1cfd"
server_key_version: "encryptionkey@0"
I20260925 05:06:53.459204 10528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:06:53.462613 10543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:53.463994 10528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 05:06:53.464169 10528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "c5671cb02dfc4145bfc344b88006fa9e"
format_stamp: "Formatted at 2026-09-25 05:06:53 on dist-test-slave-wpdm"
server_key: "9435227bc46d736b5958352e1d8116b2"
server_key_iv: "02ee66169f8c4154b822a70dc49a1cfd"
server_key_version: "encryptionkey@0"
I20260925 05:06:53.464385 10528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:53.489411 10528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:53.494117 10528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:53.494371 10528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:53.497229 10385 heartbeater.cc:503] Master 127.8.215.62:33227 was elected leader, sending a full tablet report...
I20260925 05:06:53.497453 10528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:53.497566 10528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:53.497728 10528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:53.497861 10528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 05:06:53 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:53.525640 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.498606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.8.215.3:38401)
0925 05:06:53.499621 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.499683 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.502412 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.503303 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.503317 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.504393 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:53.505570 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.505614 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.507024 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.507033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.509361 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.509374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.510398 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.510406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.510849 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.511795 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:53.511818 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:53.515506 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:53.520446 (+ 4940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:53.520462 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:53.520484 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:53.521147 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:53.521801 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:53.521807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:53.521812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:53.522152 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:53.522932 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:53.522941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:53.523490 (+ 549us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.523872 (+ 382us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.524306 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":413,"threads_started":1}
I20260925 05:06:53.525803 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.498132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:38401 (local address 127.8.215.62:33227)
0925 05:06:53.498384 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.498393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.499817 (+ 1424us) server_negotiation.cc:407] Connection header received
0925 05:06:53.502760 (+ 2943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.502768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.502862 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.502972 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:53.505816 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.506783 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.509522 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.510025 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.510294 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.515864 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:53.515873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:53.515881 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:53.515928 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:53.520175 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.521317 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.521326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.521333 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.521535 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.522348 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.522356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.522361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.522794 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:53.522956 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.524984 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.525307 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:06:53.549595 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.532936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:32855 (local address 127.8.215.62:33227)
0925 05:06:53.533159 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.533168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.533835 (+ 667us) server_negotiation.cc:407] Connection header received
0925 05:06:53.533966 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.533974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.534073 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.534192 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:53.535144 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.536606 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.538704 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.539643 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.539810 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.542575 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:53.542586 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:53.542591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:53.542637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:53.546355 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.547358 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.547366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.547372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.547548 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.547964 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.547970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.547973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.548370 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:53.548519 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.549005 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.549225 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 05:06:53.549705 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.532969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.8.215.3:32855)
0925 05:06:53.533712 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.533736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.533866 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.534244 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.534247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.534673 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:53.535009 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.535019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.536818 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.536827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.538542 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.538553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.539834 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.539838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.539947 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.540648 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:53.540685 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:53.542304 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:53.546533 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:53.546541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:53.546545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:53.547222 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:53.547700 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:53.547705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:53.547708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:53.547837 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:53.548486 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:53.548490 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:53.548693 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.548897 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.549251 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":427,"threads_started":1}
I20260925 05:06:53.556398 10662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39465 every 8 connection(s)
I20260925 05:06:53.556478 10528 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39465
I20260925 05:06:53.560441 10528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:53.562738 10663 heartbeater.cc:347] Connected to a master server at 127.8.215.62:33227
I20260925 05:06:53.562933 10663 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:53.563503 10663 heartbeater.cc:511] Master 127.8.215.62:33227 requested a full tablet report, sending...
I20260925 05:06:53.564841 10178 ts_manager.cc:194] Registered new tserver with Master: c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3:39465)
I20260925 05:06:53.565619 10178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:38401
I20260925 05:06:53.569247 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 10528
I20260925 05:06:53.569440 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:53.569866 9052 external_mini_cluster.cc:1428] Setting key be1f0851ee47594173721f0437ab3c98
I20260925 05:06:53.572866 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:06:53 dist-test-slave-wpdm krb5kdc[10133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:06:53 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:53.619175 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.605797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.8.215.62:33227)
0925 05:06:53.605973 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.605978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.606298 (+ 320us) server_negotiation.cc:407] Connection header received
0925 05:06:53.606450 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.606454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.606517 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.606608 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:53.607961 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.608702 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.610220 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.610672 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.610806 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.614596 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:53.614601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:53.614604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:53.614636 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:53.617011 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.617582 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.617586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.617588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.617704 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:53.618009 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:53.618012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:53.618014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:53.618303 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:53.618509 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.618710 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.618862 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:06:53.623179 10178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:53.629966 10178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:53.657302 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.642053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:40285 (local address 127.0.0.1:45970)
0925 05:06:53.643836 (+ 1783us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.643891 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.644119 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.646035 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.646045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.646078 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:53.646589 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.646608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.649156 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.649165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.652538 (+ 3373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.652561 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.655666 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.655676 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.656002 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.656085 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.656280 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.656581 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"mutex_wait_us":67,"thread_start_us":2781,"threads_started":1}
I20260925 05:06:53.658404 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.642344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39465 (local address 127.0.0.1:37448)
0925 05:06:53.645093 (+ 2749us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.645111 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.645220 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.647437 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.647445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.647481 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:53.648014 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.648028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.650796 (+ 2768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.650805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.654081 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.654104 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.657268 (+ 3164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.657277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.657521 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.657549 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.657629 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.657811 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2618,"mutex_wait_us":5,"thread_start_us":1106,"threads_started":1}
I20260925 05:06:53.659132 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.642258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.8.215.2:40285)
0925 05:06:53.644740 (+ 2482us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.644753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.644788 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:06:53.644898 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.644907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.645582 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.645793 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:53.646855 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.648929 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.652821 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.654968 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.655149 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.655479 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.658462 (+ 2983us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.658734 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2308,"thread_start_us":2204,"threads_started":1}
I20260925 05:06:53.659329 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.643189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.8.215.3:39465)
0925 05:06:53.646344 (+ 3155us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.646368 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.646404 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:06:53.646504 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.646513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.646785 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.647020 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:53.648296 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.650569 (+ 2273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.654362 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.656533 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.656707 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.657050 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.658669 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.658887 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2999,"thread_start_us":2828,"threads_started":1}
I20260925 05:06:53.663008 10454 tablet_service.cc:1467] Processing CreateTablet for tablet 816034af0f2241db94052a392d93b487 (DEFAULT_TABLE table=test-table [id=bffa60647f48412fb001936487e95fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:53.662863 10593 tablet_service.cc:1467] Processing CreateTablet for tablet 816034af0f2241db94052a392d93b487 (DEFAULT_TABLE table=test-table [id=bffa60647f48412fb001936487e95fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:53.664754 10593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816034af0f2241db94052a392d93b487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:53.664875 10454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816034af0f2241db94052a392d93b487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:53.667833 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.646688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.8.215.1:36539)
0925 05:06:53.659744 (+ 13056us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.659754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.659785 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:06:53.659879 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.659887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.660161 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.660934 (+ 773us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:53.661137 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.662670 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.665010 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.666685 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.666836 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.667043 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.667324 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.667505 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12888,"thread_start_us":13759,"threads_started":1}
I20260925 05:06:53.668411 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.646167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:36539 (local address 127.0.0.1:35780)
0925 05:06:53.658056 (+ 11889us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.658082 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.658225 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.660515 (+ 2290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.660520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.660551 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:53.660750 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.660761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.662868 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.662873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.664785 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.664803 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.666887 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.666894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.667080 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.667121 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.667226 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.667414 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11740,"thread_start_us":13634,"threads_started":1}
I20260925 05:06:53.673211 10315 tablet_service.cc:1467] Processing CreateTablet for tablet 816034af0f2241db94052a392d93b487 (DEFAULT_TABLE table=test-table [id=bffa60647f48412fb001936487e95fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:53.674875 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816034af0f2241db94052a392d93b487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:53.678700 10682 tablet_bootstrap.cc:492] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Bootstrap starting.
I20260925 05:06:53.681766 10683 tablet_bootstrap.cc:492] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Bootstrap starting.
I20260925 05:06:53.683466 10682 tablet_bootstrap.cc:654] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:53.684881 10682 log.cc:824] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:53.685667 10683 tablet_bootstrap.cc:654] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:53.687415 10683 log.cc:824] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:53.693022 10686 tablet_bootstrap.cc:492] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Bootstrap starting.
I20260925 05:06:53.695034 10683 tablet_bootstrap.cc:492] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: No bootstrap required, opened a new log
I20260925 05:06:53.695313 10682 tablet_bootstrap.cc:492] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: No bootstrap required, opened a new log
I20260925 05:06:53.695362 10683 ts_tablet_manager.cc:1397] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260925 05:06:53.695593 10682 ts_tablet_manager.cc:1397] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260925 05:06:53.697000 10686 tablet_bootstrap.cc:654] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:53.698328 10686 log.cc:824] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:53.703893 10686 tablet_bootstrap.cc:492] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: No bootstrap required, opened a new log
I20260925 05:06:53.704216 10686 ts_tablet_manager.cc:1397] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 05:06:53.704312 10683 raft_consensus.cc:348] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.704838 10683 raft_consensus.cc:374] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:53.704969 10683 raft_consensus.cc:729] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5671cb02dfc4145bfc344b88006fa9e, State: Initialized, Role: FOLLOWER
I20260925 05:06:53.705612 10683 consensus_queue.cc:260] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.706823 10663 heartbeater.cc:503] Master 127.8.215.62:33227 was elected leader, sending a full tablet report...
I20260925 05:06:53.707526 10683 ts_tablet_manager.cc:1428] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260925 05:06:53.706564 10682 raft_consensus.cc:348] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.708441 10682 raft_consensus.cc:374] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:53.708592 10682 raft_consensus.cc:729] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b52894ac33f4c0d943958ee5374551b, State: Initialized, Role: FOLLOWER
I20260925 05:06:53.709403 10682 consensus_queue.cc:260] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.712321 10682 ts_tablet_manager.cc:1428] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260925 05:06:53.712618 10524 heartbeater.cc:503] Master 127.8.215.62:33227 was elected leader, sending a full tablet report...
I20260925 05:06:53.714775 10686 raft_consensus.cc:348] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.715289 10686 raft_consensus.cc:374] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:53.715395 10686 raft_consensus.cc:729] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30fc178d88e4986a54cbd7104a14abe, State: Initialized, Role: FOLLOWER
I20260925 05:06:53.716094 10686 consensus_queue.cc:260] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.717865 10686 ts_tablet_manager.cc:1428] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260925 05:06:53.740398 10386 tablet.cc:2369] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:06:53.772109 10525 tablet.cc:2369] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:06:53.812296 10664 tablet.cc:2369] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:53.966393 10688 raft_consensus.cc:482] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:53.966769 10688 raft_consensus.cc:504] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.968626 10688 leader_election.cc:288] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285), a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539)
I20260925 05:06:53.975745 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.968838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:40285 (local address 127.8.215.3:33671)
0925 05:06:53.969014 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.969041 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.969201 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.969672 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.969676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.969702 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:53.969942 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.969954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.971281 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.971285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.973446 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.973462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.975047 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.975052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.975226 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.975253 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.975342 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.975447 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:06:53.975817 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.969101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:33671 (local address 127.8.215.2:40285)
0925 05:06:53.969325 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.969330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.969352 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:06:53.969434 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.969442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.969537 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.969647 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:53.970108 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.971125 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.973620 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.974699 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.975009 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.975156 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.975401 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.975544 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:06:53.976497 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.969340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:42423 (local address 127.8.215.1:36539)
0925 05:06:53.969554 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:53.969563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:53.969967 (+ 404us) server_negotiation.cc:407] Connection header received
0925 05:06:53.970118 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:53.970122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:53.970212 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:53.970308 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:53.970786 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.971627 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.974102 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.975536 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.975684 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.975796 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:53.976187 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 05:06:53.976333 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:53.976639 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:53.969171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:36539 (local address 127.8.215.3:42423)
0925 05:06:53.969843 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:53.969868 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:53.969989 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:53.970329 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:53.970332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:53.970350 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:53.970620 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.970627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.972018 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.972023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.973942 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:53.973952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:53.975806 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:53.975814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:53.976021 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:53.976058 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:06:53.976156 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:06:53.976302 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":371,"threads_started":1}
I20260925 05:06:53.977183 10479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816034af0f2241db94052a392d93b487" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b52894ac33f4c0d943958ee5374551b" is_pre_election: true
I20260925 05:06:53.977728 10479 raft_consensus.cc:2454] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 0.
I20260925 05:06:53.977785 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816034af0f2241db94052a392d93b487" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30fc178d88e4986a54cbd7104a14abe" is_pre_election: true
I20260925 05:06:53.978436 10335 raft_consensus.cc:2454] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 0.
I20260925 05:06:53.978435 10545 leader_election.cc:302] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b52894ac33f4c0d943958ee5374551b, c5671cb02dfc4145bfc344b88006fa9e; no voters:
I20260925 05:06:53.978886 10688 raft_consensus.cc:2781] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:53.979049 10688 raft_consensus.cc:482] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:53.979176 10688 raft_consensus.cc:3037] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:53.981521 10688 raft_consensus.cc:504] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.982203 10688 leader_election.cc:288] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 election: Requested vote from peers 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285), a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539)
I20260925 05:06:53.982823 10479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816034af0f2241db94052a392d93b487" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b52894ac33f4c0d943958ee5374551b"
I20260925 05:06:53.982824 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816034af0f2241db94052a392d93b487" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30fc178d88e4986a54cbd7104a14abe"
I20260925 05:06:53.983043 10479 raft_consensus.cc:3037] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:53.983043 10335 raft_consensus.cc:3037] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:53.986464 10335 raft_consensus.cc:2454] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 1.
I20260925 05:06:53.986464 10479 raft_consensus.cc:2454] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 1.
I20260925 05:06:53.986965 10546 leader_election.cc:302] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30fc178d88e4986a54cbd7104a14abe, c5671cb02dfc4145bfc344b88006fa9e; no voters:
I20260925 05:06:53.987285 10688 raft_consensus.cc:2781] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:53.987742 10688 raft_consensus.cc:686] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 1 LEADER]: Becoming Leader. State: Replica: c5671cb02dfc4145bfc344b88006fa9e, State: Running, Role: LEADER
I20260925 05:06:53.988229 10688 consensus_queue.cc:237] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } }
I20260925 05:06:53.993196 10177 catalog_manager.cc:5495] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3). New cstate: current_term: 1 leader_uuid: "c5671cb02dfc4145bfc344b88006fa9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } health_report { overall_health: UNKNOWN } } }
I20260925 05:06:54.042383 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.037129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.8.215.3:39465)
0925 05:06:54.037314 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.037318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.037458 (+ 140us) server_negotiation.cc:407] Connection header received
0925 05:06:54.037750 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.037754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.037841 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.037966 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:54.038262 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.038949 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.040256 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.040658 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.040849 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.041251 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.041794 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.041901 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.042005 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.042075 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:06:54.050961 10479 raft_consensus.cc:1261] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Refusing update from remote peer c5671cb02dfc4145bfc344b88006fa9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:54.050959 10335 raft_consensus.cc:1261] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Refusing update from remote peer c5671cb02dfc4145bfc344b88006fa9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:54.051985 10692 consensus_queue.cc:1035] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:54.052361 10688 consensus_queue.cc:1035] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:54.064465 10695 mvcc.cc:205] Tried to move back new op lower bound from 7333121286342012928 to 7333121286101086208. Current Snapshot: MvccSnapshot[applied={T|T < 7333121286342012928}]
I20260925 05:06:54.066335 10696 mvcc.cc:205] Tried to move back new op lower bound from 7333121286342012928 to 7333121286101086208. Current Snapshot: MvccSnapshot[applied={T|T < 7333121286342012928}]
I20260925 05:06:54.083809 10178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:54.084088 10178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:54.087777 10178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: Sending DeleteTablet for 3 replicas of tablet 816034af0f2241db94052a392d93b487
I20260925 05:06:54.089062 10593 tablet_service.cc:1514] Processing DeleteTablet for tablet 816034af0f2241db94052a392d93b487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:37448
I20260925 05:06:54.089062 10315 tablet_service.cc:1514] Processing DeleteTablet for tablet 816034af0f2241db94052a392d93b487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:35780
I20260925 05:06:54.089195 10454 tablet_service.cc:1514] Processing DeleteTablet for tablet 816034af0f2241db94052a392d93b487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:45970
I20260925 05:06:54.091208 10178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:54.092236 10178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:54.094221 10706 tablet_replica.cc:330] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: stopping tablet replica
I20260925 05:06:54.094749 10706 raft_consensus.cc:2229] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:54.095409 10706 raft_consensus.cc:2258] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:54.096239 10705 tablet_replica.cc:330] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: stopping tablet replica
I20260925 05:06:54.096746 10705 raft_consensus.cc:2229] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:54.097348 10705 raft_consensus.cc:2258] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:54.098201 10704 tablet_replica.cc:330] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: stopping tablet replica
I20260925 05:06:54.098258 10706 ts_tablet_manager.cc:1905] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:54.098775 10704 raft_consensus.cc:2229] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:54.099398 10704 raft_consensus.cc:2258] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:54.099926 10705 ts_tablet_manager.cc:1905] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:54.102532 10704 ts_tablet_manager.cc:1905] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:54.104489 10315 tablet_service.cc:1467] Processing CreateTablet for tablet cdd334ee732446a18f245f9f017b6e8c (DEFAULT_TABLE table=test-table [id=e2c5c0bb4c0a4f16a11b6296e3ff4198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:54.105005 10454 tablet_service.cc:1467] Processing CreateTablet for tablet cdd334ee732446a18f245f9f017b6e8c (DEFAULT_TABLE table=test-table [id=e2c5c0bb4c0a4f16a11b6296e3ff4198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:54.105472 10454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd334ee732446a18f245f9f017b6e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:54.105885 10706 ts_tablet_manager.cc:1918] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:54.106038 10706 log.cc:1196] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/816034af0f2241db94052a392d93b487
I20260925 05:06:54.106699 10706 ts_tablet_manager.cc:1939] T 816034af0f2241db94052a392d93b487 P 6b52894ac33f4c0d943958ee5374551b: Deleting consensus metadata
I20260925 05:06:54.107101 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd334ee732446a18f245f9f017b6e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:54.107604 10593 tablet_service.cc:1467] Processing CreateTablet for tablet cdd334ee732446a18f245f9f017b6e8c (DEFAULT_TABLE table=test-table [id=e2c5c0bb4c0a4f16a11b6296e3ff4198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:54.108031 10593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd334ee732446a18f245f9f017b6e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:54.108788 10163 catalog_manager.cc:4841] TS 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285): tablet 816034af0f2241db94052a392d93b487 (table test-table [id=bffa60647f48412fb001936487e95fa3]) successfully deleted
I20260925 05:06:54.110934 10704 ts_tablet_manager.cc:1918] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:54.111145 10704 log.cc:1196] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/816034af0f2241db94052a392d93b487
I20260925 05:06:54.111753 10704 ts_tablet_manager.cc:1939] T 816034af0f2241db94052a392d93b487 P c5671cb02dfc4145bfc344b88006fa9e: Deleting consensus metadata
I20260925 05:06:54.113648 10165 catalog_manager.cc:4841] TS c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3:39465): tablet 816034af0f2241db94052a392d93b487 (table test-table [id=bffa60647f48412fb001936487e95fa3]) successfully deleted
I20260925 05:06:54.116711 10705 ts_tablet_manager.cc:1918] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:54.116963 10705 log.cc:1196] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/816034af0f2241db94052a392d93b487
I20260925 05:06:54.117599 10705 ts_tablet_manager.cc:1939] T 816034af0f2241db94052a392d93b487 P a30fc178d88e4986a54cbd7104a14abe: Deleting consensus metadata
I20260925 05:06:54.119179 10683 tablet_bootstrap.cc:492] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Bootstrap starting.
I20260925 05:06:54.120886 10683 tablet_bootstrap.cc:654] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:54.121886 10682 tablet_bootstrap.cc:492] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Bootstrap starting.
I20260925 05:06:54.122466 10164 catalog_manager.cc:4841] TS a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539): tablet 816034af0f2241db94052a392d93b487 (table test-table [id=bffa60647f48412fb001936487e95fa3]) successfully deleted
I20260925 05:06:54.123541 10682 tablet_bootstrap.cc:654] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:54.124545 10683 tablet_bootstrap.cc:492] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: No bootstrap required, opened a new log
I20260925 05:06:54.124718 10683 ts_tablet_manager.cc:1397] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260925 05:06:54.125584 10683 raft_consensus.cc:348] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.125818 10683 raft_consensus.cc:374] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:54.125931 10683 raft_consensus.cc:729] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5671cb02dfc4145bfc344b88006fa9e, State: Initialized, Role: FOLLOWER
I20260925 05:06:54.126267 10686 tablet_bootstrap.cc:492] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Bootstrap starting.
I20260925 05:06:54.127941 10686 tablet_bootstrap.cc:654] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:54.129927 10683 consensus_queue.cc:260] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.130913 10683 ts_tablet_manager.cc:1428] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260925 05:06:54.132320 10686 tablet_bootstrap.cc:492] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: No bootstrap required, opened a new log
I20260925 05:06:54.132462 10686 ts_tablet_manager.cc:1397] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 05:06:54.133288 10686 raft_consensus.cc:348] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.133507 10686 raft_consensus.cc:374] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:54.133595 10686 raft_consensus.cc:729] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30fc178d88e4986a54cbd7104a14abe, State: Initialized, Role: FOLLOWER
I20260925 05:06:54.133926 10686 consensus_queue.cc:260] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.134675 10686 ts_tablet_manager.cc:1428] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:06:54.138181 10682 tablet_bootstrap.cc:492] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: No bootstrap required, opened a new log
I20260925 05:06:54.138334 10682 ts_tablet_manager.cc:1397] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260925 05:06:54.138996 10682 raft_consensus.cc:348] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.139163 10682 raft_consensus.cc:374] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:54.139273 10682 raft_consensus.cc:729] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b52894ac33f4c0d943958ee5374551b, State: Initialized, Role: FOLLOWER
I20260925 05:06:54.139550 10682 consensus_queue.cc:260] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.140105 10682 ts_tablet_manager.cc:1428] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:06:54.165191 10688 raft_consensus.cc:482] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:54.165400 10688 raft_consensus.cc:504] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.166149 10688 leader_election.cc:288] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539), 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285)
I20260925 05:06:54.166774 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd334ee732446a18f245f9f017b6e8c" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30fc178d88e4986a54cbd7104a14abe" is_pre_election: true
I20260925 05:06:54.166827 10479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd334ee732446a18f245f9f017b6e8c" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b52894ac33f4c0d943958ee5374551b" is_pre_election: true
I20260925 05:06:54.167023 10335 raft_consensus.cc:2454] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 0.
I20260925 05:06:54.167109 10479 raft_consensus.cc:2454] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 0.
I20260925 05:06:54.167513 10546 leader_election.cc:302] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30fc178d88e4986a54cbd7104a14abe, c5671cb02dfc4145bfc344b88006fa9e; no voters:
I20260925 05:06:54.167794 10688 raft_consensus.cc:2781] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:54.167930 10688 raft_consensus.cc:482] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:54.168022 10688 raft_consensus.cc:3037] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:54.171377 10688 raft_consensus.cc:504] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.172103 10688 leader_election.cc:288] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 election: Requested vote from peers a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539), 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285)
I20260925 05:06:54.172585 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd334ee732446a18f245f9f017b6e8c" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30fc178d88e4986a54cbd7104a14abe"
I20260925 05:06:54.172839 10335 raft_consensus.cc:3037] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:54.172938 10479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd334ee732446a18f245f9f017b6e8c" candidate_uuid: "c5671cb02dfc4145bfc344b88006fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b52894ac33f4c0d943958ee5374551b"
I20260925 05:06:54.173170 10479 raft_consensus.cc:3037] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:54.175150 10479 raft_consensus.cc:2454] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 1.
I20260925 05:06:54.175616 10545 leader_election.cc:302] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b52894ac33f4c0d943958ee5374551b, c5671cb02dfc4145bfc344b88006fa9e; no voters:
I20260925 05:06:54.175944 10688 raft_consensus.cc:2781] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:54.175969 10335 raft_consensus.cc:2454] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5671cb02dfc4145bfc344b88006fa9e in term 1.
I20260925 05:06:54.176110 10688 raft_consensus.cc:686] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 1 LEADER]: Becoming Leader. State: Replica: c5671cb02dfc4145bfc344b88006fa9e, State: Running, Role: LEADER
I20260925 05:06:54.176409 10688 consensus_queue.cc:237] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.179421 10177 catalog_manager.cc:5495] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3). New cstate: current_term: 1 leader_uuid: "c5671cb02dfc4145bfc344b88006fa9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } health_report { overall_health: HEALTHY } } }
Sep 25 05:06:54 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:54.224438 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.207299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33227 (local address 127.0.0.1:45456)
0925 05:06:54.208007 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:54.208033 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:54.208208 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:54.208676 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:54.208682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:54.209683 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:54.209951 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.209961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.211322 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.211329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.213017 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.213034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.213834 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.213841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.214037 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.214786 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:54.214818 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:54.217572 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:54.221769 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:54.221782 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:54.221788 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:54.222307 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:54.222761 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:54.222764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:54.222767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:54.222905 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:54.223543 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:54.223550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:54.223704 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 05:06:54.223850 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 05:06:54.224075 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":420,"threads_started":1}
I20260925 05:06:54.224750 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.207336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.8.215.62:33227)
0925 05:06:54.208373 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.208376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.208392 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:06:54.208431 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.208434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.208482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.208615 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:54.210130 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.211167 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.213213 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.213638 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.213754 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.217830 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:54.217838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:54.217847 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:54.217888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:54.221581 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.222472 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.222478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.222482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.222648 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.223019 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.223025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.223028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.223393 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:54.223537 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.223993 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.224231 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":793,"threads_started":1}
I20260925 05:06:54.230540 10177 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:45456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:06:54.231828 10177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:54.238672 10315 tablet_service.cc:1467] Processing CreateTablet for tablet ae466a6260864b3e8d5665126094e10a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10e267d03c8544f196f22fd75c137513]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:54.238870 10454 tablet_service.cc:1467] Processing CreateTablet for tablet ae466a6260864b3e8d5665126094e10a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10e267d03c8544f196f22fd75c137513]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:54.239066 10593 tablet_service.cc:1467] Processing CreateTablet for tablet ae466a6260864b3e8d5665126094e10a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10e267d03c8544f196f22fd75c137513]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:54.239257 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae466a6260864b3e8d5665126094e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:54.239411 10454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae466a6260864b3e8d5665126094e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:54.239583 10593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae466a6260864b3e8d5665126094e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:54.246605 10686 tablet_bootstrap.cc:492] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: Bootstrap starting.
I20260925 05:06:54.246639 10682 tablet_bootstrap.cc:492] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: Bootstrap starting.
I20260925 05:06:54.248327 10682 tablet_bootstrap.cc:654] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:54.248389 10686 tablet_bootstrap.cc:654] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:54.249437 10683 tablet_bootstrap.cc:492] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: Bootstrap starting.
I20260925 05:06:54.251178 10682 tablet_bootstrap.cc:492] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: No bootstrap required, opened a new log
I20260925 05:06:54.251204 10683 tablet_bootstrap.cc:654] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:54.251369 10682 ts_tablet_manager.cc:1397] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:06:54.252318 10682 raft_consensus.cc:348] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.252574 10682 raft_consensus.cc:374] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:54.252671 10682 raft_consensus.cc:729] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b52894ac33f4c0d943958ee5374551b, State: Initialized, Role: FOLLOWER
I20260925 05:06:54.253082 10682 consensus_queue.cc:260] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.253662 10682 ts_tablet_manager.cc:1428] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:06:54.253546 10689 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.253921 10689 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.255146 10686 tablet_bootstrap.cc:492] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: No bootstrap required, opened a new log
I20260925 05:06:54.256065 10686 ts_tablet_manager.cc:1397] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260925 05:06:54.256873 10686 raft_consensus.cc:348] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.257117 10686 raft_consensus.cc:374] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:54.257201 10686 raft_consensus.cc:729] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30fc178d88e4986a54cbd7104a14abe, State: Initialized, Role: FOLLOWER
I20260925 05:06:54.257444 10686 consensus_queue.cc:260] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.258028 10686 ts_tablet_manager.cc:1428] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:54.257911 10690 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.258309 10690 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.262332 10683 tablet_bootstrap.cc:492] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: No bootstrap required, opened a new log
I20260925 05:06:54.262530 10683 ts_tablet_manager.cc:1397] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260925 05:06:54.263379 10683 raft_consensus.cc:348] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.263630 10683 raft_consensus.cc:374] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:54.263713 10683 raft_consensus.cc:729] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5671cb02dfc4145bfc344b88006fa9e, State: Initialized, Role: FOLLOWER
I20260925 05:06:54.263963 10683 consensus_queue.cc:260] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.264420 10688 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.264711 10688 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.265276 10683 ts_tablet_manager.cc:1428] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 05:06:54.318979 10690 raft_consensus.cc:482] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:54.319229 10690 raft_consensus.cc:504] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.320847 10690 leader_election.cc:288] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285), c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3:39465)
I20260925 05:06:54.329218 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.321406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:44675 (local address 127.8.215.3:39465)
0925 05:06:54.321666 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.321676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.322667 (+ 991us) server_negotiation.cc:407] Connection header received
0925 05:06:54.322885 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.322892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.322967 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.323068 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:54.323780 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.324963 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.326969 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.328162 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.328348 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.328448 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.328758 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.328899 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:06:54.329265 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.321123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39465 (local address 127.8.215.1:44675)
0925 05:06:54.322582 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:54.322599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:54.322749 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:54.323128 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:54.323134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:54.323161 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:54.323594 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.323605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.325090 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.325098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.326803 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.326815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.328383 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.328389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.328551 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.328585 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:06:54.328678 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:06:54.328821 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":25,"thread_start_us":755,"threads_started":1}
I20260925 05:06:54.330148 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.321557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:57581 (local address 127.8.215.2:40285)
0925 05:06:54.321733 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.321739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.322377 (+ 638us) server_negotiation.cc:407] Connection header received
0925 05:06:54.322716 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.322722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.322821 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.322927 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:54.323512 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.324696 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.327421 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.328709 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.328844 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.328937 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.329753 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.329876 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:54.330885 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.321080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:40285 (local address 127.8.215.1:57581)
0925 05:06:54.322246 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:54.322269 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:54.322503 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:54.322945 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:54.322950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:54.322974 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:54.323354 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.323364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.324847 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.324852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.327223 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.327237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.329082 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.329088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.329565 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.329597 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:06:54.330316 (+ 719us) client_negotiation.cc:240] Negotiation successful
0925 05:06:54.330455 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":107,"thread_start_us":1197,"threads_started":1}
I20260925 05:06:54.330587 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae466a6260864b3e8d5665126094e10a" candidate_uuid: "a30fc178d88e4986a54cbd7104a14abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5671cb02dfc4145bfc344b88006fa9e" is_pre_election: true
I20260925 05:06:54.331159 10613 raft_consensus.cc:2454] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a30fc178d88e4986a54cbd7104a14abe in term 0.
I20260925 05:06:54.331678 10479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae466a6260864b3e8d5665126094e10a" candidate_uuid: "a30fc178d88e4986a54cbd7104a14abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b52894ac33f4c0d943958ee5374551b" is_pre_election: true
I20260925 05:06:54.331832 10268 leader_election.cc:302] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30fc178d88e4986a54cbd7104a14abe, c5671cb02dfc4145bfc344b88006fa9e; no voters:
I20260925 05:06:54.331940 10479 raft_consensus.cc:2454] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a30fc178d88e4986a54cbd7104a14abe in term 0.
I20260925 05:06:54.332229 10690 raft_consensus.cc:2781] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:54.332356 10690 raft_consensus.cc:482] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:54.332459 10690 raft_consensus.cc:3037] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:54.335001 10690 raft_consensus.cc:504] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.335762 10690 leader_election.cc:288] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [CANDIDATE]: Term 1 election: Requested vote from peers 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285), c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3:39465)
I20260925 05:06:54.336160 10479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae466a6260864b3e8d5665126094e10a" candidate_uuid: "a30fc178d88e4986a54cbd7104a14abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b52894ac33f4c0d943958ee5374551b"
I20260925 05:06:54.336421 10479 raft_consensus.cc:3037] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:54.336403 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae466a6260864b3e8d5665126094e10a" candidate_uuid: "a30fc178d88e4986a54cbd7104a14abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5671cb02dfc4145bfc344b88006fa9e"
I20260925 05:06:54.336611 10613 raft_consensus.cc:3037] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:54.339406 10479 raft_consensus.cc:2454] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a30fc178d88e4986a54cbd7104a14abe in term 1.
I20260925 05:06:54.339902 10266 leader_election.cc:302] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b52894ac33f4c0d943958ee5374551b, a30fc178d88e4986a54cbd7104a14abe; no voters:
I20260925 05:06:54.340247 10690 raft_consensus.cc:2781] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:54.340235 10613 raft_consensus.cc:2454] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a30fc178d88e4986a54cbd7104a14abe in term 1.
I20260925 05:06:54.341302 10690 raft_consensus.cc:686] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [term 1 LEADER]: Becoming Leader. State: Replica: a30fc178d88e4986a54cbd7104a14abe, State: Running, Role: LEADER
I20260925 05:06:54.341742 10690 consensus_queue.cc:237] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } }
I20260925 05:06:54.343969 10723 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: TxnStatusTablet state changed. Reason: New leader a30fc178d88e4986a54cbd7104a14abe. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.344807 10723 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:54.345595 10725 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:06:54.345813 10725 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:06:54.346216 10177 catalog_manager.cc:5495] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe reported cstate change: term changed from 0 to 1, leader changed from <none> to a30fc178d88e4986a54cbd7104a14abe (127.8.215.1). New cstate: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260925 05:06:54.367255 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.359437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:36539 (local address 127.0.0.1:35792)
0925 05:06:54.359839 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:54.359861 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:54.360021 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:54.360652 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:54.360659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:54.360681 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:54.360918 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.360932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.362486 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.362492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.364104 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.364119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.365155 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.365162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.365366 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.365443 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.366562 (+ 1119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.366594 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:06:54.366741 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 05:06:54.366942 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260925 05:06:54.367249 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.359437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.8.215.1:36539)
0925 05:06:54.359620 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.359626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.359962 (+ 336us) server_negotiation.cc:407] Connection header received
0925 05:06:54.360234 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.360240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.360415 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.360614 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:54.361068 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.362330 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.364260 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.364984 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.365128 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.365664 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.366434 (+ 770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.366632 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.366791 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.366892 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:54.372119 10479 raft_consensus.cc:1261] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Refusing update from remote peer a30fc178d88e4986a54cbd7104a14abe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:54.372366 10613 raft_consensus.cc:1261] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Refusing update from remote peer a30fc178d88e4986a54cbd7104a14abe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:54.373253 10690 consensus_queue.cc:1035] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:54.373742 10723 consensus_queue.cc:1035] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:54.389205 10689 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: TxnStatusTablet state changed. Reason: New leader a30fc178d88e4986a54cbd7104a14abe. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.389528 10689 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.393514 10723 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.393779 10723 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:54.394022 10697 mvcc.cc:205] Tried to move back new op lower bound from 7333121287660576768 to 7333121287547817984. Current Snapshot: MvccSnapshot[applied={T|T < 7333121287660576768 or (T in {7333121287660576768})}]
I20260925 05:06:54.394731 10689 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.394958 10689 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P 6b52894ac33f4c0d943958ee5374551b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.396807 10690 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.397101 10690 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P a30fc178d88e4986a54cbd7104a14abe: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:54.397254 10688 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: TxnStatusTablet state changed. Reason: New leader a30fc178d88e4986a54cbd7104a14abe. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.397578 10688 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.405283 10688 tablet_replica.cc:439] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a30fc178d88e4986a54cbd7104a14abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 } } peers { permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 } } peers { permanent_uuid: "c5671cb02dfc4145bfc344b88006fa9e" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39465 } } }
I20260925 05:06:54.405581 10688 tablet_replica.cc:442] T ae466a6260864b3e8d5665126094e10a P c5671cb02dfc4145bfc344b88006fa9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:54.426429 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.419063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:57495 (local address 127.8.215.1:36539)
0925 05:06:54.419285 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.419291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.419456 (+ 165us) server_negotiation.cc:407] Connection header received
0925 05:06:54.419779 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.419786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.419871 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.419969 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:54.420714 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.421956 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.424196 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.424895 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.425039 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.425584 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.425750 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.425856 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.426011 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.426108 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:06:54.426574 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.418554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:36539 (local address 127.8.215.3:57495)
0925 05:06:54.419347 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:54.419378 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:54.419603 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:54.420005 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:54.420025 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:54.420041 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:54.420519 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.420532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.422104 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.422111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.424039 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.424054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.425100 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.425107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.425309 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.425381 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.425868 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.425881 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:06:54.425975 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:06:54.426128 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":741,"threads_started":1}
I20260925 05:06:54.438510 10478 raft_consensus.cc:1261] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Refusing update from remote peer c5671cb02dfc4145bfc344b88006fa9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:54.438621 10335 raft_consensus.cc:1261] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Refusing update from remote peer c5671cb02dfc4145bfc344b88006fa9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:54.439278 10692 consensus_queue.cc:1035] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30fc178d88e4986a54cbd7104a14abe" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:54.439752 10692 consensus_queue.cc:1035] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b52894ac33f4c0d943958ee5374551b" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:54.523783 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.511339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:40591 (local address 127.8.215.3:39465)
0925 05:06:54.511489 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.511495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.517339 (+ 5844us) server_negotiation.cc:407] Connection header received
0925 05:06:54.517704 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.517710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.517794 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.517861 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:54.518796 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.519707 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.521414 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.522132 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.522223 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.522966 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.523114 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.523177 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.523401 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.523486 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:06:54.525040 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.510845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39465 (local address 127.8.215.1:40591)
0925 05:06:54.517216 (+ 6371us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:54.517244 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:54.517469 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:54.518094 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:54.518099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:54.518110 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:54.518615 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.518628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.519843 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.519846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.521260 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:54.521273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.522412 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.522418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:54.522697 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.522772 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:54.523259 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:54.523269 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:06:54.524376 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0925 05:06:54.524531 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6129,"thread_start_us":5968,"threads_started":1}
I20260925 05:06:54.603058 10177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:54.603367 10177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:54.607426 10177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c198cd0d50174f15aa889f5f28e32fd8: Sending DeleteTablet for 3 replicas of tablet cdd334ee732446a18f245f9f017b6e8c
I20260925 05:06:54.608551 10454 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd334ee732446a18f245f9f017b6e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:45970
I20260925 05:06:54.608899 10706 tablet_replica.cc:330] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: stopping tablet replica
I20260925 05:06:54.609125 10706 raft_consensus.cc:2229] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:54.609346 10706 raft_consensus.cc:2258] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:54.610265 10593 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd334ee732446a18f245f9f017b6e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:37448
I20260925 05:06:54.610626 10704 tablet_replica.cc:330] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: stopping tablet replica
I20260925 05:06:54.610863 10704 raft_consensus.cc:2229] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:54.611246 10704 raft_consensus.cc:2258] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:54.611608 10706 ts_tablet_manager.cc:1905] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:54.613963 10704 ts_tablet_manager.cc:1905] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:54.618894 10706 ts_tablet_manager.cc:1918] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:54.619040 10706 log.cc:1196] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/cdd334ee732446a18f245f9f017b6e8c
I20260925 05:06:54.621846 10315 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd334ee732446a18f245f9f017b6e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:35780
I20260925 05:06:54.627827 10706 ts_tablet_manager.cc:1939] T cdd334ee732446a18f245f9f017b6e8c P 6b52894ac33f4c0d943958ee5374551b: Deleting consensus metadata
I20260925 05:06:54.629760 10740 tablet_replica.cc:330] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: stopping tablet replica
I20260925 05:06:54.630039 10740 raft_consensus.cc:2229] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:54.630031 10163 catalog_manager.cc:4841] TS 6b52894ac33f4c0d943958ee5374551b (127.8.215.2:40285): tablet cdd334ee732446a18f245f9f017b6e8c (table test-table [id=e2c5c0bb4c0a4f16a11b6296e3ff4198]) successfully deleted
I20260925 05:06:54.630364 10740 raft_consensus.cc:2258] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:54.633865 10740 ts_tablet_manager.cc:1905] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:54.636214 10704 ts_tablet_manager.cc:1918] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:54.636446 10704 log.cc:1196] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/cdd334ee732446a18f245f9f017b6e8c
I20260925 05:06:54.636927 10704 ts_tablet_manager.cc:1939] T cdd334ee732446a18f245f9f017b6e8c P c5671cb02dfc4145bfc344b88006fa9e: Deleting consensus metadata
I20260925 05:06:54.638136 10165 catalog_manager.cc:4841] TS c5671cb02dfc4145bfc344b88006fa9e (127.8.215.3:39465): tablet cdd334ee732446a18f245f9f017b6e8c (table test-table [id=e2c5c0bb4c0a4f16a11b6296e3ff4198]) successfully deleted
I20260925 05:06:54.647257 10740 ts_tablet_manager.cc:1918] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:54.647498 10740 log.cc:1196] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/cdd334ee732446a18f245f9f017b6e8c
I20260925 05:06:54.648037 10740 ts_tablet_manager.cc:1939] T cdd334ee732446a18f245f9f017b6e8c P a30fc178d88e4986a54cbd7104a14abe: Deleting consensus metadata
I20260925 05:06:54.649278 10164 catalog_manager.cc:4841] TS a30fc178d88e4986a54cbd7104a14abe (127.8.215.1:36539): tablet cdd334ee732446a18f245f9f017b6e8c (table test-table [id=e2c5c0bb4c0a4f16a11b6296e3ff4198]) successfully deleted
Sep 25 05:06:54 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:06:54.667867 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.634386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.8.215.1:36539)
0925 05:06:54.634569 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.634575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.640661 (+ 6086us) server_negotiation.cc:407] Connection header received
0925 05:06:54.647295 (+ 6634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.647302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.647390 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.647457 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:54.649932 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.651111 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.653247 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.653999 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.654104 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.659685 (+ 5581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:54.659699 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:54.659718 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:54.659758 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:54.664211 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.665252 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.665262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.665265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.665457 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.665965 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.665973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.665978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.666574 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:54.667029 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.667244 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.667482 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 05:06:54.668810 10380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35804
I20260925 05:06:54.689282 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.672294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.8.215.62:33227)
0925 05:06:54.672527 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.672534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.672848 (+ 314us) server_negotiation.cc:407] Connection header received
0925 05:06:54.673016 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.673023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.673126 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.673232 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:54.674922 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.676156 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.678101 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.678820 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.678924 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.682396 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:54.682404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:54.682409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:54.682452 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:54.686141 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.687012 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.687018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.687022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.687189 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.687631 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.687636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.687639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.688009 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:54.688267 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.688651 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.688926 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260925 05:06:54.689918 10177 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45472
I20260925 05:06:54.691932 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10249
I20260925 05:06:54.711583 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.692945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.8.215.1:36539)
0925 05:06:54.693146 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.693153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.697344 (+ 4191us) server_negotiation.cc:407] Connection header received
0925 05:06:54.697501 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.697507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.697597 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.697707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:54.699133 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.700220 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.701936 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.702499 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.702605 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.705451 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:54.705458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:54.705461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:54.705494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:54.708382 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.709344 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.709349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.709353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.709514 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.710117 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.710122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.710125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.710461 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:54.710733 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.710923 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.711211 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 05:06:54.712218 10380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35812
W20260925 05:06:54.712812 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:54.713003 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 10249
I20260925 05:06:54.734723 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10389
Sep 25 05:06:54 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:06:54.749550 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.735565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.8.215.2:40285)
0925 05:06:54.735735 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.735740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.735761 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:06:54.735826 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.735832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.735903 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.736009 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:54.737249 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.738017 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.739242 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.739663 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.739791 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.743849 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:54.743865 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:54.743881 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:54.743940 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:54.746774 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.747440 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.747448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.747450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.747588 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.747969 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.747974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.747977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.748356 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:54.748658 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.748997 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.749221 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 05:06:54.750236 10519 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45982
W20260925 05:06:54.750871 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:54.751012 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 10389
I20260925 05:06:54.768271 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10528
Sep 25 05:06:54 dist-test-slave-wpdm krb5kdc[10133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:06:54.783377 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:54.769093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.8.215.3:39465)
0925 05:06:54.769304 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:54.769311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:54.769355 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:06:54.769567 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:54.769572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:54.769644 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:54.769760 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:54.770990 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.771705 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.773048 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:54.773481 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:54.773615 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:54.777797 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:54.777813 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:54.777829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:54.777863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:54.780597 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.781391 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.781400 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.781403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.781544 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:54.781923 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:54.781928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:54.781933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:54.782277 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:54.782588 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:54.782824 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 05:06:54.783041 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 05:06:54.784150 10658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464
W20260925 05:06:54.784821 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10528: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:54.784984 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 10528
I20260925 05:06:54.806905 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10146
I20260925 05:06:54.808022 10218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:54.922869 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 10146
2026-09-25T05:06:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4787 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:55 dist-test-slave-wpdm krb5kdc[10749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:55 dist-test-slave-wpdm krb5kdc[10749](info): set up 2 sockets
Sep 25 05:06:55 dist-test-slave-wpdm krb5kdc[10749](info): commencing operation
krb5kdc: starting...
W20260925 05:06:56.057349 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:06:56 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:06:56.244146 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:34773
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:34773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:56.557299 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:56.557653 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:56.557832 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:56.564994 10762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:56.565155 10762 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:06:56.565251 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:56.565366 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:56.565460 10762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:56.565548 10762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:56.575371 10762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:34773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:34773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:56.576705 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:56.578152 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:56.590075 10767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:56.590193 10768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:56.590160 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:56.590991 10762 server_base.cc:1034] running on GCE node
I20260925 05:06:56.591763 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:56.593155 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:56.594380 10762 hybrid_clock.cc:648] HybridClock initialized: now 1790312816594319 us; error 32 us; skew 500 ppm
I20260925 05:06:56.594719 10762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:56 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:56.597872 10762 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:06:56.599301 10762 webserver.cc:460] Webserver started at http://127.8.215.62:34293/ using document root <none> and password file <none>
I20260925 05:06:56.599992 10762 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:56.600104 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:56.600430 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:56.603425 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "3d6ae535aa464ec0910a06a81ecec850"
format_stamp: "Formatted at 2026-09-25 05:06:56 on dist-test-slave-wpdm"
server_key: "81b7066da53ce9148ab9bb8189660fee"
server_key_iv: "625bedba2c97dc58fdbee7608381f8a9"
server_key_version: "encryptionkey@0"
I20260925 05:06:56.604110 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "3d6ae535aa464ec0910a06a81ecec850"
format_stamp: "Formatted at 2026-09-25 05:06:56 on dist-test-slave-wpdm"
server_key: "81b7066da53ce9148ab9bb8189660fee"
server_key_iv: "625bedba2c97dc58fdbee7608381f8a9"
server_key_version: "encryptionkey@0"
I20260925 05:06:56.609038 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:06:56.612342 10777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:56.613737 10762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:56.613933 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "3d6ae535aa464ec0910a06a81ecec850"
format_stamp: "Formatted at 2026-09-25 05:06:56 on dist-test-slave-wpdm"
server_key: "81b7066da53ce9148ab9bb8189660fee"
server_key_iv: "625bedba2c97dc58fdbee7608381f8a9"
server_key_version: "encryptionkey@0"
I20260925 05:06:56.614151 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:56.643282 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:56.647307 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:56.647580 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:56.669051 10762 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:34773
I20260925 05:06:56.669070 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:34773 every 8 connection(s)
I20260925 05:06:56.670441 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:56.672027 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 10762
I20260925 05:06:56.672235 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:06:56.672911 9052 external_mini_cluster.cc:1428] Setting key ab9d2c478f16c33ea09391aba34c25c4
I20260925 05:06:56.674799 10839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:06:56 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:56.688355 10839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Bootstrap starting.
I20260925 05:06:56.692591 10839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:56.693893 10839 log.cc:824] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:56.697664 10839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: No bootstrap required, opened a new log
I20260925 05:06:56.697726 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:56.675042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.8.215.62:34773)
0925 05:06:56.675823 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:56.675838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:56.675899 (+ 61us) server_negotiation.cc:407] Connection header received
0925 05:06:56.678007 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:56.678036 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:56.678578 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:56.679047 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:56.680331 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:56.682475 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:56.683827 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:56.684880 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:56.685072 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:56.688924 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:56.688934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:56.688955 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:56.689002 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:56.692941 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:56.693593 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:56.693604 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:56.693614 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:56.693847 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:56.694243 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:56.694249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:56.694252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:56.695014 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:56.695300 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:56.695844 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 05:06:56.696270 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":325,"threads_started":1}
I20260925 05:06:56.706655 10839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6ae535aa464ec0910a06a81ecec850" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34773 } }
I20260925 05:06:56.707011 10839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:56.707106 10839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6ae535aa464ec0910a06a81ecec850, State: Initialized, Role: FOLLOWER
I20260925 05:06:56.707612 10839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6ae535aa464ec0910a06a81ecec850" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34773 } }
I20260925 05:06:56.707814 10839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:56.707916 10839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:56.708066 10839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:56.710304 10839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6ae535aa464ec0910a06a81ecec850" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34773 } }
I20260925 05:06:56.710821 10839 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6ae535aa464ec0910a06a81ecec850; no voters:
I20260925 05:06:56.711354 10839 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:56.711501 10844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:56.712756 10844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6ae535aa464ec0910a06a81ecec850, State: Running, Role: LEADER
I20260925 05:06:56.713333 10844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6ae535aa464ec0910a06a81ecec850" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34773 } }
I20260925 05:06:56.713712 10839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:56.716940 10846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6ae535aa464ec0910a06a81ecec850. Latest consensus state: current_term: 1 leader_uuid: "3d6ae535aa464ec0910a06a81ecec850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6ae535aa464ec0910a06a81ecec850" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34773 } } }
I20260925 05:06:56.717355 10846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:56.716883 10845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6ae535aa464ec0910a06a81ecec850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6ae535aa464ec0910a06a81ecec850" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34773 } } }
I20260925 05:06:56.717988 10845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:56.722047 10853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:56.726459 10853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:56.735488 10853 catalog_manager.cc:1347] Generated new cluster ID: 454cae8e732e48859e2ffd0ac24e7385
I20260925 05:06:56.735610 10853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:56.745445 10853 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:56.747331 10853 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:56.759509 10853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Generated new TSK 0
I20260925 05:06:56.760555 10853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:06:56.809949 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:34773
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:57.136850 10865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:57.137207 10865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:57.137383 10865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:57.144840 10865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:57.145000 10865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:57.145153 10865 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:06:57.145247 10865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:57.145382 10865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:06:57.155378 10865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:34773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:57.156777 10865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:57.158143 10865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:57.171303 10870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:57.171202 10873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:57.171248 10871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:57.172128 10865 server_base.cc:1034] running on GCE node
I20260925 05:06:57.172851 10865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:57.173836 10865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:57.175071 10865 hybrid_clock.cc:648] HybridClock initialized: now 1790312817175014 us; error 83 us; skew 500 ppm
I20260925 05:06:57.175407 10865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:57 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312817, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:57.178746 10865 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:06:57.180419 10865 webserver.cc:460] Webserver started at http://127.8.215.1:34199/ using document root <none> and password file <none>
I20260925 05:06:57.181169 10865 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:57.181317 10865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:57.181633 10865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:57.184670 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "673a3092070944a7a155e67df40ebffa"
format_stamp: "Formatted at 2026-09-25 05:06:57 on dist-test-slave-wpdm"
server_key: "82235a711f4faa6acefb5cee7dcfbc3c"
server_key_iv: "acecdef6ac3e15b415f05600bbe61042"
server_key_version: "encryptionkey@0"
I20260925 05:06:57.185504 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "673a3092070944a7a155e67df40ebffa"
format_stamp: "Formatted at 2026-09-25 05:06:57 on dist-test-slave-wpdm"
server_key: "82235a711f4faa6acefb5cee7dcfbc3c"
server_key_iv: "acecdef6ac3e15b415f05600bbe61042"
server_key_version: "encryptionkey@0"
I20260925 05:06:57.190630 10865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:06:57.194034 10880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:57.195335 10865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:06:57.195514 10865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "673a3092070944a7a155e67df40ebffa"
format_stamp: "Formatted at 2026-09-25 05:06:57 on dist-test-slave-wpdm"
server_key: "82235a711f4faa6acefb5cee7dcfbc3c"
server_key_iv: "acecdef6ac3e15b415f05600bbe61042"
server_key_version: "encryptionkey@0"
I20260925 05:06:57.195719 10865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:57.215080 10865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:57.219691 10865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:57.219990 10865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:57.222755 10865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:57.222898 10865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:57.223048 10865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:57.223146 10865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:06:57 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312817, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:57.245285 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.223484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:39647 (local address 127.8.215.62:34773)
0925 05:06:57.225845 (+ 2361us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:57.225852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:57.225878 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:06:57.227458 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:57.227463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:57.227560 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:57.227699 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:57.230032 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.231121 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.233805 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.234233 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.234370 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.239021 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:57.239027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:57.239031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:57.239070 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:57.241929 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.242695 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.242706 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.242709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.242820 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.243263 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.243267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.243271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.243552 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:57.243725 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:57.244639 (+ 914us) server_negotiation.cc:299] Negotiation successful
0925 05:06:57.244912 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2231,"thread_start_us":2135,"threads_started":1}
I20260925 05:06:57.246897 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.224092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.8.215.1:39647)
0925 05:06:57.225062 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:57.225112 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:57.227154 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:57.227934 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:57.227943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:57.228778 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:57.229840 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.229873 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.231305 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.231312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.233641 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.233657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.234504 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.234514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.234770 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.235584 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:57.235615 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:57.238731 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:57.242078 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.242087 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.242115 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.242561 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.242932 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.242937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.242941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.243145 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.243695 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:57.243702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:57.244064 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 05:06:57.244559 (+ 495us) client_negotiation.cc:240] Negotiation successful
0925 05:06:57.245155 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":363,"threads_started":1}
I20260925 05:06:57.269210 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.253928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:58053 (local address 127.8.215.62:34773)
0925 05:06:57.254084 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:57.254088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:57.254811 (+ 723us) server_negotiation.cc:407] Connection header received
0925 05:06:57.255036 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:57.255040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:57.255121 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:57.255230 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:57.256991 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.257693 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.260115 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.260560 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.260688 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.262923 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:57.262928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:57.262931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:57.262959 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:57.265839 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.266883 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.266891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.266895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.267050 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.267519 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.267526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.267530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.267960 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:57.268148 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:57.268601 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 05:06:57.268859 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:06:57.269395 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.253928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.8.215.1:58053)
0925 05:06:57.254697 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:57.254713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:57.254889 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:57.255292 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:57.255303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:57.256175 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:57.256787 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.256809 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.257850 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.257858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.259958 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.259977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.260802 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.260814 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.261101 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.261987 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:57.262020 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:57.262742 (+ 722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:57.266006 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.266018 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.266024 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.266713 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.267171 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.267177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.267181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.267374 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.268111 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:57.268117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:57.268329 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 05:06:57.268502 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 05:06:57.268884 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":433,"threads_started":1}
I20260925 05:06:57.283783 10865 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:42311
I20260925 05:06:57.283787 11000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:42311 every 8 connection(s)
I20260925 05:06:57.285269 10865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:57.287540 11001 heartbeater.cc:347] Connected to a master server at 127.8.215.62:34773
I20260925 05:06:57.287801 11001 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:57.288457 11001 heartbeater.cc:511] Master 127.8.215.62:34773 requested a full tablet report, sending...
I20260925 05:06:57.290890 10794 ts_manager.cc:194] Registered new tserver with Master: 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311)
I20260925 05:06:57.291493 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 10865
I20260925 05:06:57.291704 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:57.292325 9052 external_mini_cluster.cc:1428] Setting key a809705b35658040e4d176c457e59616
I20260925 05:06:57.292824 10794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:39647
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:06:57.336551 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:34773
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:57.680233 11005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:57.680553 11005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:57.680796 11005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:57.688271 11005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:57.688427 11005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:57.688599 11005 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:06:57.688687 11005 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:57.688813 11005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:06:57.698892 11005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:34773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:57.700233 11005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:57.701668 11005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:57.714885 11010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:57.714917 11013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:57.714833 11011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:57.715935 11005 server_base.cc:1034] running on GCE node
I20260925 05:06:57.716563 11005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:57.717811 11005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:57.719044 11005 hybrid_clock.cc:648] HybridClock initialized: now 1790312817718940 us; error 89 us; skew 500 ppm
I20260925 05:06:57.719437 11005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:57 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312817, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:57.722594 11005 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:06:57.723941 11005 webserver.cc:460] Webserver started at http://127.8.215.2:38035/ using document root <none> and password file <none>
I20260925 05:06:57.724642 11005 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:57.724797 11005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:57.725065 11005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:57.727819 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "b47de446fed44560b9190ecb40169a28"
format_stamp: "Formatted at 2026-09-25 05:06:57 on dist-test-slave-wpdm"
server_key: "1cb481c95dd55113b7a07ccc0fedc457"
server_key_iv: "8f23d6f95737296b39f26778f9fd9619"
server_key_version: "encryptionkey@0"
I20260925 05:06:57.728510 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "b47de446fed44560b9190ecb40169a28"
format_stamp: "Formatted at 2026-09-25 05:06:57 on dist-test-slave-wpdm"
server_key: "1cb481c95dd55113b7a07ccc0fedc457"
server_key_iv: "8f23d6f95737296b39f26778f9fd9619"
server_key_version: "encryptionkey@0"
I20260925 05:06:57.733657 11005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:06:57.737277 11020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:57.738977 11005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 05:06:57.739148 11005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "b47de446fed44560b9190ecb40169a28"
format_stamp: "Formatted at 2026-09-25 05:06:57 on dist-test-slave-wpdm"
server_key: "1cb481c95dd55113b7a07ccc0fedc457"
server_key_iv: "8f23d6f95737296b39f26778f9fd9619"
server_key_version: "encryptionkey@0"
I20260925 05:06:57.739377 11005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:57.766261 11005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:57.770243 11005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:57.770526 11005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:57.773096 11005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:57.773224 11005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:57.773360 11005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:57.773449 11005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:06:57 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312817, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:57.797608 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.773801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:38943 (local address 127.8.215.62:34773)
0925 05:06:57.774234 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:57.774240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:57.775471 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 05:06:57.778449 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:57.778458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:57.778565 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:57.778690 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:57.781495 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.782460 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.785052 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.785758 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.785883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.790473 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:57.790479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:57.790486 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:57.790530 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:57.793870 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.795032 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.795038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.795042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.795220 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.795786 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.795806 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.795809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.796080 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:57.796234 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:57.797000 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 05:06:57.797213 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":206,"threads_started":1}
I20260925 05:06:57.799362 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.774392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.8.215.2:38943)
0925 05:06:57.775273 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:57.775332 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:57.778090 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:57.779010 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:57.779026 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:57.780059 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:57.781248 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.781291 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.782678 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.782688 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.784896 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.784908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.786093 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.786100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.786272 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.786990 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:57.787022 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:57.790186 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:57.793985 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.793997 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.794019 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.794839 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.795356 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.795362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.795365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.795650 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.796219 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:57.796227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:57.796578 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 05:06:57.796899 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 05:06:57.797504 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":315,"threads_started":1}
I20260925 05:06:57.823068 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.806855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50727 (local address 127.8.215.62:34773)
0925 05:06:57.807086 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:57.807096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:57.807664 (+ 568us) server_negotiation.cc:407] Connection header received
0925 05:06:57.807862 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:57.807866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:57.807926 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:57.808005 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:57.809736 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.811008 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.813608 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.814016 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.814159 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.816280 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:57.816285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:57.816288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:57.816319 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:57.819733 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.820374 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.820380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.820384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.820509 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:57.820943 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:57.820948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:57.820952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:57.821422 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:57.821583 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:57.822671 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0925 05:06:57.822866 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:06:57.823194 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:57.806806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.8.215.2:50727)
0925 05:06:57.807539 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:57.807564 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:57.807739 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:57.808072 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:57.808079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:57.808853 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:57.809529 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.809549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.811243 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.811251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.813467 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:57.813483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:57.814268 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:57.814276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:57.814477 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:57.815212 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:57.815228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:57.816093 (+ 865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:57.819869 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.819874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.819879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.820255 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.820616 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:57.820622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:57.820625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:57.820822 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:57.821552 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:57.821557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:57.821863 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 05:06:57.822052 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 05:06:57.822417 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":371,"threads_started":1}
I20260925 05:06:57.827721 11005 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:32943
I20260925 05:06:57.827737 11140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:32943 every 8 connection(s)
I20260925 05:06:57.828984 11005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:57.830845 11141 heartbeater.cc:347] Connected to a master server at 127.8.215.62:34773
I20260925 05:06:57.831081 11141 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:57.831687 11141 heartbeater.cc:511] Master 127.8.215.62:34773 requested a full tablet report, sending...
I20260925 05:06:57.833036 10793 ts_manager.cc:194] Registered new tserver with Master: b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:57.833884 10793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:38943
I20260925 05:06:57.836395 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 11005
I20260925 05:06:57.836596 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:57.837141 9052 external_mini_cluster.cc:1428] Setting key 369eabe377ff7b399d8a56e625c7ee7d
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:06:57.880247 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:34773
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:58.199339 11145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:58.199617 11145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:58.199792 11145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:58.207047 11145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:58.207152 11145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:58.207235 11145 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:06:58.207279 11145 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:58.207348 11145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:06:58.217495 11145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:34773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:58.218955 11145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:58.220247 11145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:58.233546 11153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:58.233497 11151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:58.233541 11150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:58.234791 11145 server_base.cc:1034] running on GCE node
I20260925 05:06:58.235384 11145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:58.236292 11145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:58.237532 11145 hybrid_clock.cc:648] HybridClock initialized: now 1790312818237470 us; error 55 us; skew 500 ppm
I20260925 05:06:58.237855 11145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:58 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:58.244374 11145 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:06:58.246001 11145 webserver.cc:460] Webserver started at http://127.8.215.3:46761/ using document root <none> and password file <none>
I20260925 05:06:58.246709 11145 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:58.246853 11145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:58.247205 11145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:58.250344 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "e69f42ffe4db4b2c8530946d3d44e93a"
format_stamp: "Formatted at 2026-09-25 05:06:58 on dist-test-slave-wpdm"
server_key: "caa863bb4a1ce8631c57ad864de52b11"
server_key_iv: "88911be28fa9ed961359545e32cfadb0"
server_key_version: "encryptionkey@0"
I20260925 05:06:58.251191 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "e69f42ffe4db4b2c8530946d3d44e93a"
format_stamp: "Formatted at 2026-09-25 05:06:58 on dist-test-slave-wpdm"
server_key: "caa863bb4a1ce8631c57ad864de52b11"
server_key_iv: "88911be28fa9ed961359545e32cfadb0"
server_key_version: "encryptionkey@0"
I20260925 05:06:58.256395 11145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:06:58.259945 11160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:58.261288 11145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:58.261462 11145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "e69f42ffe4db4b2c8530946d3d44e93a"
format_stamp: "Formatted at 2026-09-25 05:06:58 on dist-test-slave-wpdm"
server_key: "caa863bb4a1ce8631c57ad864de52b11"
server_key_iv: "88911be28fa9ed961359545e32cfadb0"
server_key_version: "encryptionkey@0"
I20260925 05:06:58.261673 11145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:58.293717 11145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:58.296464 11001 heartbeater.cc:503] Master 127.8.215.62:34773 was elected leader, sending a full tablet report...
I20260925 05:06:58.297776 11145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:58.298118 11145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:58.301143 11145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:58.301263 11145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:58.301380 11145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:58.301491 11145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:06:58 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:58.323711 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.301651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:36753 (local address 127.8.215.62:34773)
0925 05:06:58.301836 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.301841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.303367 (+ 1526us) server_negotiation.cc:407] Connection header received
0925 05:06:58.305367 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.305372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.305469 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.305588 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:58.307698 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.309021 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.311702 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.312281 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.312420 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.317330 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:58.317335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:58.317339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:58.317369 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:58.320096 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:58.320847 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:58.320854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:58.320858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:58.320988 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:58.321450 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:58.321454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:58.321458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:58.321740 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:58.321898 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.322905 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.323185 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:06:58.324846 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.302346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.8.215.3:36753)
0925 05:06:58.303221 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.303256 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.305087 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.305817 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.305825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.306592 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:58.307465 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.307494 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.309220 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.309229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.311542 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.311556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.312580 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.312588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.312845 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.313630 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:58.313654 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:58.317077 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:58.320252 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:58.320262 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:58.320279 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:58.320692 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:58.321104 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:58.321109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:58.321113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:58.321322 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:58.321883 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:58.321891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:58.322410 (+ 519us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.322756 (+ 346us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.323403 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":355,"threads_started":1}
I20260925 05:06:58.347903 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.331865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.8.215.3:39925)
0925 05:06:58.332897 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.332913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.333021 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.333361 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.333364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.333749 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:58.334044 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.334052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.335694 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.335703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.337961 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.337978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.339197 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.339202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.339356 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.340005 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:58.340022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:58.340706 (+ 684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:58.345069 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:58.345079 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:58.345083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:58.345499 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:58.345904 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:58.345909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:58.345913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:58.346038 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:58.346745 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:58.346748 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:58.347114 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.347315 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.347577 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":654,"threads_started":1}
I20260925 05:06:58.348328 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.331869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:39925 (local address 127.8.215.62:34773)
0925 05:06:58.332081 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.332091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.333001 (+ 910us) server_negotiation.cc:407] Connection header received
0925 05:06:58.333123 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.333131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.333228 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.333338 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:58.334182 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.335486 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.338179 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.338973 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.339125 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.340950 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:58.340956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:58.340961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:58.340999 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:58.344925 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:58.345636 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:58.345641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:58.345645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:58.345789 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:58.346156 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:58.346161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:58.346164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:58.346546 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:58.346687 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.347503 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.347817 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:06:58.355996 11145 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:36305
I20260925 05:06:58.356035 11279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:36305 every 8 connection(s)
I20260925 05:06:58.357568 11145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:58.359179 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 11145
I20260925 05:06:58.359359 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:58.359870 9052 external_mini_cluster.cc:1428] Setting key e08249916036c249367d87ac67cf013b
I20260925 05:06:58.359812 11280 heartbeater.cc:347] Connected to a master server at 127.8.215.62:34773
I20260925 05:06:58.360695 11280 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:58.361584 11280 heartbeater.cc:511] Master 127.8.215.62:34773 requested a full tablet report, sending...
I20260925 05:06:58.363006 10794 ts_manager.cc:194] Registered new tserver with Master: e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305)
I20260925 05:06:58.363770 10794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:36753
I20260925 05:06:58.375463 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:06:58 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:06:58 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:58.422462 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.408506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.8.215.62:34773)
0925 05:06:58.408765 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.408774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.409079 (+ 305us) server_negotiation.cc:407] Connection header received
0925 05:06:58.409227 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.409232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.409300 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.409404 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:58.410715 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.411545 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.412811 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.413259 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.413371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.417473 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:58.417478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:58.417481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:58.417512 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:58.420147 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:58.420777 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:58.420780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:58.420783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:58.420903 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:58.421290 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:58.421293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:58.421296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:58.421541 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:58.421730 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.421986 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.422143 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:06:58.426680 10794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:58.433699 10794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:58.476881 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.445847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.0.0.1:34096)
0925 05:06:58.460227 (+ 14380us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.460275 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.460446 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.463171 (+ 2725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.463183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.463226 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.463778 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.463795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.467248 (+ 3453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.467257 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.471449 (+ 4192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.471471 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.475091 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.475100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.475439 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.475596 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.475977 (+ 381us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.476274 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14071,"threads_started":1}
I20260925 05:06:58.479794 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.446390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.8.215.3:36305)
0925 05:06:58.461795 (+ 15405us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.461808 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.461855 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:06:58.461983 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.461995 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.462371 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.462692 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.464338 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.467007 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.471916 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.474318 (+ 2402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.474486 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.474916 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.479011 (+ 4095us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.479333 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15129,"thread_start_us":17561,"threads_started":1}
I20260925 05:06:58.485400 11210 tablet_service.cc:1467] Processing CreateTablet for tablet 262182f3977c4e0d855b50cc12214eac (DEFAULT_TABLE table=test-table [id=1a0ba417081a4fe48ed70c253ee24e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:58.487368 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.448580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.0.0.1:55050)
0925 05:06:58.477186 (+ 28606us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.477209 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.477336 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.478618 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.478635 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.478759 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.479631 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.479646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.481690 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.481707 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.483426 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.483438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.486318 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.486326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.486626 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.486685 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.486788 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.486918 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28456,"mutex_wait_us":649,"spinlock_wait_cycles":42240,"thread_start_us":28133,"threads_started":1}
I20260925 05:06:58.487993 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.449352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.8.215.1:42311)
0925 05:06:58.475275 (+ 25923us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.475288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.477481 (+ 2193us) server_negotiation.cc:407] Connection header received
0925 05:06:58.477644 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.477657 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.478082 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.478341 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.479874 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.481475 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.483613 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.485374 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.485707 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.486118 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.487473 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.487653 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25683,"thread_start_us":25960,"threads_started":1}
I20260925 05:06:58.488005 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262182f3977c4e0d855b50cc12214eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:58.488303 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.446844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.8.215.2:32943)
0925 05:06:58.477585 (+ 30741us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.477595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.477648 (+ 53us) server_negotiation.cc:407] Connection header received
0925 05:06:58.477879 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.477890 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.478266 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.478541 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.479572 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.481505 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.484626 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.486897 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.487007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.487200 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.487779 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.487958 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30505,"thread_start_us":30358,"threads_started":1}
I20260925 05:06:58.488299 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.446212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.0.0.1:47342)
0925 05:06:58.476404 (+ 30192us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.476435 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.476597 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.478532 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.478537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.478561 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.479222 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.479252 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.481732 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.481737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.484295 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.484310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.487360 (+ 3050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.487369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.487573 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.487606 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.487734 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.487859 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29970,"mutex_wait_us":1,"thread_start_us":13553,"threads_started":1}
I20260925 05:06:58.492864 10931 tablet_service.cc:1467] Processing CreateTablet for tablet 262182f3977c4e0d855b50cc12214eac (DEFAULT_TABLE table=test-table [id=1a0ba417081a4fe48ed70c253ee24e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:58.492872 11071 tablet_service.cc:1467] Processing CreateTablet for tablet 262182f3977c4e0d855b50cc12214eac (DEFAULT_TABLE table=test-table [id=1a0ba417081a4fe48ed70c253ee24e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:58.494563 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262182f3977c4e0d855b50cc12214eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:58.494554 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262182f3977c4e0d855b50cc12214eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:58.503239 11299 tablet_bootstrap.cc:492] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Bootstrap starting.
I20260925 05:06:58.507279 11299 tablet_bootstrap.cc:654] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:58.508584 11299 log.cc:824] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:58.512491 11299 tablet_bootstrap.cc:492] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: No bootstrap required, opened a new log
I20260925 05:06:58.512497 11301 tablet_bootstrap.cc:492] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Bootstrap starting.
I20260925 05:06:58.512873 11299 ts_tablet_manager.cc:1397] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 05:06:58.517463 11302 tablet_bootstrap.cc:492] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Bootstrap starting.
I20260925 05:06:58.517560 11301 tablet_bootstrap.cc:654] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:58.519372 11301 log.cc:824] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:58.521286 11302 tablet_bootstrap.cc:654] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:58.522300 11302 log.cc:824] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:58.523000 11301 tablet_bootstrap.cc:492] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: No bootstrap required, opened a new log
I20260925 05:06:58.523264 11301 ts_tablet_manager.cc:1397] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 05:06:58.524945 11302 tablet_bootstrap.cc:492] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: No bootstrap required, opened a new log
I20260925 05:06:58.525213 11302 ts_tablet_manager.cc:1397] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 05:06:58.525622 11299 raft_consensus.cc:348] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.526080 11299 raft_consensus.cc:374] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:58.526167 11299 raft_consensus.cc:729] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Initialized, Role: FOLLOWER
I20260925 05:06:58.526901 11299 consensus_queue.cc:260] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.532465 11302 raft_consensus.cc:348] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.532838 11302 raft_consensus.cc:374] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:58.532967 11302 raft_consensus.cc:729] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47de446fed44560b9190ecb40169a28, State: Initialized, Role: FOLLOWER
I20260925 05:06:58.533578 11280 heartbeater.cc:503] Master 127.8.215.62:34773 was elected leader, sending a full tablet report...
I20260925 05:06:58.534296 11299 ts_tablet_manager.cc:1428] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260925 05:06:58.534811 11302 consensus_queue.cc:260] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.535912 11141 heartbeater.cc:503] Master 127.8.215.62:34773 was elected leader, sending a full tablet report...
I20260925 05:06:58.536464 11302 ts_tablet_manager.cc:1428] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260925 05:06:58.537549 11301 raft_consensus.cc:348] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.537976 11301 raft_consensus.cc:374] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:58.538098 11301 raft_consensus.cc:729] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Initialized, Role: FOLLOWER
I20260925 05:06:58.538776 11301 consensus_queue.cc:260] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.540416 11301 ts_tablet_manager.cc:1428] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260925 05:06:58.581004 11142 tablet.cc:2369] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:06:58.609206 11281 tablet.cc:2369] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:06:58.787868 11002 tablet.cc:2369] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:58.896898 11307 raft_consensus.cc:482] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:58.897251 11307 raft_consensus.cc:504] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.899039 11307 leader_election.cc:288] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305), b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:58.903602 11306 raft_consensus.cc:482] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:58.903930 11306 raft_consensus.cc:504] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.905977 11306 leader_election.cc:288] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305), 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311)
I20260925 05:06:58.907387 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.899493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:50901 (local address 127.8.215.3:36305)
0925 05:06:58.899681 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.899686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.900528 (+ 842us) server_negotiation.cc:407] Connection header received
0925 05:06:58.900788 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.900796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.900885 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.900964 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.901772 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.902492 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.904755 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.905943 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.906067 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.906183 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.906886 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.907093 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:06:58.908593 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.899449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.8.215.1:50901)
0925 05:06:58.900428 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.900447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.900576 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.901038 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.901042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.901081 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.901610 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.901622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.902618 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.902623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.904577 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.904589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.906385 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.906407 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.906642 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.906682 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.907918 (+ 1236us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.908116 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":433,"threads_started":1}
I20260925 05:06:58.909896 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262182f3977c4e0d855b50cc12214eac" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" is_pre_election: true
I20260925 05:06:58.910480 11235 raft_consensus.cc:2454] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 0.
I20260925 05:06:58.911147 10884 leader_election.cc:302] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:06:58.911542 11307 raft_consensus.cc:2781] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:58.911713 11307 raft_consensus.cc:482] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:58.911842 11307 raft_consensus.cc:3037] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:58.914541 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.899478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:56445 (local address 127.8.215.2:32943)
0925 05:06:58.899779 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.899788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.900437 (+ 649us) server_negotiation.cc:407] Connection header received
0925 05:06:58.900643 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.900653 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.900733 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.900849 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.901602 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.902510 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.905077 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.906363 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.906535 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.906677 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.914117 (+ 7440us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.914268 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 05:06:58.915771 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.899356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.8.215.1:56445)
0925 05:06:58.900301 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.900323 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.900515 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.900890 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.900905 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.900930 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.901436 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.901446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.902700 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.902703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.904894 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.904908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.913202 (+ 8294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.913209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.913405 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.913443 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.914725 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.914875 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":2,"spinlock_wait_cycles":21760,"thread_start_us":701,"threads_started":1}
I20260925 05:06:58.916218 11307 raft_consensus.cc:504] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.917444 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262182f3977c4e0d855b50cc12214eac" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a"
I20260925 05:06:58.917665 11235 raft_consensus.cc:3037] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:58.917651 11091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262182f3977c4e0d855b50cc12214eac" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28" is_pre_election: true
I20260925 05:06:58.918246 11307 leader_election.cc:288] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 election: Requested vote from peers e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305), b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:58.918577 11091 raft_consensus.cc:2454] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 0.
I20260925 05:06:58.919274 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262182f3977c4e0d855b50cc12214eac" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28"
I20260925 05:06:58.919516 11090 raft_consensus.cc:3037] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:58.920674 11235 raft_consensus.cc:2454] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 1.
I20260925 05:06:58.921110 10884 leader_election.cc:302] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:06:58.921414 11307 raft_consensus.cc:2781] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:58.923555 11090 raft_consensus.cc:2454] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 1.
I20260925 05:06:58.924238 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.906857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:53159 (local address 127.8.215.3:36305)
0925 05:06:58.907868 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.907876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.916931 (+ 9055us) server_negotiation.cc:407] Connection header received
0925 05:06:58.917237 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.917243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.917345 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.917409 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.918892 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.920111 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.922338 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.923260 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.923417 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.923519 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.923827 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.923948 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":9225,"threads_started":1}
I20260925 05:06:58.924247 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.906297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.8.215.2:53159)
0925 05:06:58.916819 (+ 10522us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.916844 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.917063 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.917726 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.917734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.917765 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.918718 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.918729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.920265 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.920270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.922180 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.922192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.923488 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.923493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.923646 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.923694 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.923784 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.923916 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10286,"thread_start_us":10025,"threads_started":1}
I20260925 05:06:58.925051 11307 raft_consensus.cc:686] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 1 LEADER]: Becoming Leader. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Running, Role: LEADER
I20260925 05:06:58.925031 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262182f3977c4e0d855b50cc12214eac" candidate_uuid: "b47de446fed44560b9190ecb40169a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" is_pre_election: true
I20260925 05:06:58.925419 11235 raft_consensus.cc:2379] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b47de446fed44560b9190ecb40169a28 in current term 1: Already voted for candidate 673a3092070944a7a155e67df40ebffa in this term.
I20260925 05:06:58.926076 11307 consensus_queue.cc:237] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } }
I20260925 05:06:58.930361 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.906454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.8.215.2:50733)
0925 05:06:58.917077 (+ 10623us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:58.917095 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:58.917295 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:58.917847 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:58.917851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:58.917895 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:58.924317 (+ 6422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.924335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.925961 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.925966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.927963 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:58.927977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.929565 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.929573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:58.929755 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.929790 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:06:58.929893 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:06:58.930027 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10485,"thread_start_us":10258,"threads_started":1}
I20260925 05:06:58.930274 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:58.907167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50733 (local address 127.8.215.1:42311)
0925 05:06:58.907374 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:58.907382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:58.917162 (+ 9780us) server_negotiation.cc:407] Connection header received
0925 05:06:58.917484 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:58.917488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:58.917567 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:58.917631 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:58.924546 (+ 6915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.925786 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.928158 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:58.929305 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:58.929476 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:58.929609 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:58.929936 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 05:06:58.930051 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:06:58.931463 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262182f3977c4e0d855b50cc12214eac" candidate_uuid: "b47de446fed44560b9190ecb40169a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa" is_pre_election: true
I20260925 05:06:58.931516 10793 catalog_manager.cc:5495] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 673a3092070944a7a155e67df40ebffa (127.8.215.1). New cstate: current_term: 1 leader_uuid: "673a3092070944a7a155e67df40ebffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:58.932423 11021 leader_election.cc:302] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b47de446fed44560b9190ecb40169a28; no voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a
I20260925 05:06:58.932911 11306 raft_consensus.cc:2726] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 25 05:06:59 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:06:59.060173 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.047038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:34773 (local address 127.0.0.1:35522)
0925 05:06:59.047814 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.047830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.047957 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.048256 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.048259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.048995 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:59.049277 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.049293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.050408 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.050413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.051507 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.051518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.052237 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.052242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.052351 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.053065 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:59.053086 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:59.055221 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:59.057975 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:59.057985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:59.057991 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:59.058428 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:59.058729 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:59.058732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:59.058734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:59.058862 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:59.059400 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:59.059407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:59.059553 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.059679 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.059877 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":455,"threads_started":1}
I20260925 05:06:59.060318 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.046984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.8.215.62:34773)
0925 05:06:59.047673 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.047677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.047913 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:06:59.048052 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.048056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.048117 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.048214 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:59.049435 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.050215 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.051629 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.052033 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.052126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.055408 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:59.055413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:59.055416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:59.055442 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:59.057859 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:59.058527 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:59.058531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:59.058534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:59.058642 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:59.058948 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:59.058951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:59.058953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:59.059269 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:59.059437 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.059775 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.059973 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":570,"thread_start_us":377,"threads_started":1}
I20260925 05:06:59.065161 10793 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:35522:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:06:59.066321 10793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:59.072619 11071 tablet_service.cc:1467] Processing CreateTablet for tablet a991e14ba9a9485a84ee78fc61509d41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac9c3e8387294c8c8a14b19412871286]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:59.072619 10931 tablet_service.cc:1467] Processing CreateTablet for tablet a991e14ba9a9485a84ee78fc61509d41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac9c3e8387294c8c8a14b19412871286]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:59.072948 11210 tablet_service.cc:1467] Processing CreateTablet for tablet a991e14ba9a9485a84ee78fc61509d41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac9c3e8387294c8c8a14b19412871286]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:59.073625 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a991e14ba9a9485a84ee78fc61509d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:59.073625 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a991e14ba9a9485a84ee78fc61509d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:59.073738 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a991e14ba9a9485a84ee78fc61509d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:59.079663 11322 tablet_bootstrap.cc:492] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: Bootstrap starting.
I20260925 05:06:59.081238 11323 tablet_bootstrap.cc:492] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: Bootstrap starting.
I20260925 05:06:59.082401 11322 tablet_bootstrap.cc:654] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:59.082820 11324 tablet_bootstrap.cc:492] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: Bootstrap starting.
I20260925 05:06:59.083825 11323 tablet_bootstrap.cc:654] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:59.084681 11324 tablet_bootstrap.cc:654] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:59.084971 11322 tablet_bootstrap.cc:492] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: No bootstrap required, opened a new log
I20260925 05:06:59.085132 11322 ts_tablet_manager.cc:1397] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 05:06:59.086606 11323 tablet_bootstrap.cc:492] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: No bootstrap required, opened a new log
I20260925 05:06:59.086473 11322 raft_consensus.cc:348] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.086787 11323 ts_tablet_manager.cc:1397] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 05:06:59.086840 11322 raft_consensus.cc:374] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:59.086966 11322 raft_consensus.cc:729] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Initialized, Role: FOLLOWER
I20260925 05:06:59.087249 11322 consensus_queue.cc:260] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.087489 11323 raft_consensus.cc:348] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.087791 11323 raft_consensus.cc:374] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:59.087888 11323 raft_consensus.cc:729] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47de446fed44560b9190ecb40169a28, State: Initialized, Role: FOLLOWER
I20260925 05:06:59.088132 11323 consensus_queue.cc:260] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.088707 11323 ts_tablet_manager.cc:1428] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 05:06:59.088661 11306 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.089030 11306 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.090101 11322 ts_tablet_manager.cc:1428] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 05:06:59.091634 11324 tablet_bootstrap.cc:492] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: No bootstrap required, opened a new log
I20260925 05:06:59.091784 11324 ts_tablet_manager.cc:1397] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 05:06:59.092475 11328 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.092859 11328 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.092772 11324 raft_consensus.cc:348] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.093029 11324 raft_consensus.cc:374] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:59.093129 11324 raft_consensus.cc:729] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Initialized, Role: FOLLOWER
I20260925 05:06:59.093398 11324 consensus_queue.cc:260] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.093868 11307 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.094038 11324 ts_tablet_manager.cc:1428] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:06:59.094175 11307 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.120201 11328 raft_consensus.cc:482] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:59.120425 11328 raft_consensus.cc:504] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.121898 11328 leader_election.cc:288] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311), b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:59.128692 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.122318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:55575 (local address 127.8.215.2:32943)
0925 05:06:59.122486 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.122493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.123193 (+ 700us) server_negotiation.cc:407] Connection header received
0925 05:06:59.123461 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.123466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.123553 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.123653 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:59.124397 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.125184 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.126982 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.127803 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.127940 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.128035 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.128281 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.128415 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:06:59.128919 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.122307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:59347 (local address 127.8.215.1:42311)
0925 05:06:59.122486 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.122493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.123256 (+ 763us) server_negotiation.cc:407] Connection header received
0925 05:06:59.123371 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.123377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.123457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.123582 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:59.124183 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.125018 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.126998 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.127988 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.128125 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.128229 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.128492 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.128616 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:06:59.129098 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.122156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.8.215.3:55575)
0925 05:06:59.123056 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.123077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.123244 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.123823 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.123833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.123863 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:59.124240 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.124248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.125318 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.125326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.126839 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.126851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.127963 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.127969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.128129 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.128151 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.128258 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.128386 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":44,"spinlock_wait_cycles":14080,"thread_start_us":584,"threads_started":1}
I20260925 05:06:59.129228 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.122252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.8.215.3:59347)
0925 05:06:59.123056 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.123077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.123244 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.123623 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.123627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.123656 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:59.124003 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.124018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.125135 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.125143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.126855 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.126866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.128195 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.128201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.128344 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.128366 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.128468 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.128598 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":72,"spinlock_wait_cycles":22016,"thread_start_us":429,"threads_started":1}
I20260925 05:06:59.129856 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a991e14ba9a9485a84ee78fc61509d41" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa" is_pre_election: true
I20260925 05:06:59.129854 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a991e14ba9a9485a84ee78fc61509d41" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28" is_pre_election: true
I20260925 05:06:59.130138 11090 raft_consensus.cc:2454] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 0.
I20260925 05:06:59.130149 10956 raft_consensus.cc:2454] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 0.
I20260925 05:06:59.130580 11163 leader_election.cc:302] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, b47de446fed44560b9190ecb40169a28, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:06:59.131002 11328 raft_consensus.cc:2781] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:59.131124 11328 raft_consensus.cc:482] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:59.131245 11328 raft_consensus.cc:3037] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:59.133402 11328 raft_consensus.cc:504] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.134032 11328 leader_election.cc:288] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 election: Requested vote from peers 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311), b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:59.134485 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a991e14ba9a9485a84ee78fc61509d41" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa"
I20260925 05:06:59.134522 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a991e14ba9a9485a84ee78fc61509d41" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28"
I20260925 05:06:59.134725 10956 raft_consensus.cc:3037] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:59.134728 11090 raft_consensus.cc:3037] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:59.137697 10956 raft_consensus.cc:2454] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 1.
I20260925 05:06:59.137698 11090 raft_consensus.cc:2454] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 1.
I20260925 05:06:59.138104 11163 leader_election.cc:302] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b47de446fed44560b9190ecb40169a28, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:06:59.138394 11328 raft_consensus.cc:2781] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:59.138850 11328 raft_consensus.cc:686] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 LEADER]: Becoming Leader. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Running, Role: LEADER
I20260925 05:06:59.139324 11328 consensus_queue.cc:237] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.142079 11331 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: TxnStatusTablet state changed. Reason: New leader e69f42ffe4db4b2c8530946d3d44e93a. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.142498 11331 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:59.143482 11333 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:06:59.143757 11333 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:06:59.143741 10794 catalog_manager.cc:5495] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a reported cstate change: term changed from 0 to 1, leader changed from <none> to e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3). New cstate: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:59.167575 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.161470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.0.0.1:34108)
0925 05:06:59.161766 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.161788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.161958 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.162438 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.162443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.162455 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:59.162654 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.162663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.163791 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.163797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.164982 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.164993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.165858 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.165865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.166049 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.166101 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.166991 (+ 890us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.167012 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.167106 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.167268 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 05:06:59.167578 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.161580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.8.215.3:36305)
0925 05:06:59.161790 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.161797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.161883 (+ 86us) server_negotiation.cc:407] Connection header received
0925 05:06:59.162179 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.162183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.162264 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.162423 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:59.162796 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.163641 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.165126 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.165698 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.165850 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.166307 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.166872 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.167035 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.167122 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.167227 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 05:06:59.176424 11090 raft_consensus.cc:1261] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Refusing update from remote peer e69f42ffe4db4b2c8530946d3d44e93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:59.176399 10956 raft_consensus.cc:1261] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Refusing update from remote peer e69f42ffe4db4b2c8530946d3d44e93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:59.177304 11331 consensus_queue.cc:1035] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:59.177670 11328 consensus_queue.cc:1035] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:59.185312 11307 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: TxnStatusTablet state changed. Reason: New leader e69f42ffe4db4b2c8530946d3d44e93a. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.185678 11307 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.190305 11306 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: TxnStatusTablet state changed. Reason: New leader e69f42ffe4db4b2c8530946d3d44e93a. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.190475 11336 mvcc.cc:205] Tried to move back new op lower bound from 7333121307337306112 to 7333121307198582784. Current Snapshot: MvccSnapshot[applied={T|T < 7333121307337306112}]
I20260925 05:06:59.190604 11306 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.196076 11338 mvcc.cc:205] Tried to move back new op lower bound from 7333121307337306112 to 7333121307198582784. Current Snapshot: MvccSnapshot[applied={T|T < 7333121307337306112}]
I20260925 05:06:59.197505 11328 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.197816 11328 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:59.198170 11328 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.198395 11328 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P e69f42ffe4db4b2c8530946d3d44e93a: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:59.200788 11307 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.201049 11307 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P 673a3092070944a7a155e67df40ebffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.203171 11306 tablet_replica.cc:439] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } } }
I20260925 05:06:59.203462 11306 tablet_replica.cc:442] T a991e14ba9a9485a84ee78fc61509d41 P b47de446fed44560b9190ecb40169a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:59.212921 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.207027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.8.215.1:42311)
0925 05:06:59.207182 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.207186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.207752 (+ 566us) server_negotiation.cc:407] Connection header received
0925 05:06:59.207987 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.207993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.208075 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.208223 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:59.208659 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.209442 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.210816 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.211228 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.211380 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.211773 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.212281 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.212467 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.212516 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.212590 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:06:59.230726 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.223961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.8.215.1:52431)
0925 05:06:59.224656 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.224671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.224849 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.225210 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.225213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.225222 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:59.225464 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.225471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.227125 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.227129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.228353 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.228364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.229589 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.229594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.229708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.229760 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.230252 (+ 492us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.230260 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.230330 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.230426 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":398,"threads_started":1}
I20260925 05:06:59.230846 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.224008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:52431 (local address 127.8.215.3:36305)
0925 05:06:59.224214 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.224222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.224775 (+ 553us) server_negotiation.cc:407] Connection header received
0925 05:06:59.224987 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.224993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.225093 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.225202 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:59.225606 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.227001 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.228487 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.229416 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.229557 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.229989 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.230147 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.230239 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.230381 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.230477 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:06:59.240021 11090 raft_consensus.cc:1261] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Refusing update from remote peer 673a3092070944a7a155e67df40ebffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:59.240181 11235 raft_consensus.cc:1261] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Refusing update from remote peer 673a3092070944a7a155e67df40ebffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:59.240705 11307 consensus_queue.cc:1035] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:59.241025 11313 consensus_queue.cc:1035] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:59.288673 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.274853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:41113 (local address 127.8.215.1:42311)
0925 05:06:59.275097 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.275103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.281519 (+ 6416us) server_negotiation.cc:407] Connection header received
0925 05:06:59.281707 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.281715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.281807 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.281907 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:59.282743 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.283755 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.286307 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.287025 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.287145 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.287765 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.287901 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.287989 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.288212 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.288319 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 05:06:59.289695 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.274270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.8.215.3:41113)
0925 05:06:59.281389 (+ 7119us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.281410 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.281565 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.282069 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.282075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.282090 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:59.282525 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.282538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.283906 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.283913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.286095 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.286118 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.287250 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.287258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.287467 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.287550 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.288013 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.288026 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.289073 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.289243 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6895,"thread_start_us":6712,"threads_started":1}
I20260925 05:06:59.348165 10794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:59.348474 10794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:59.353049 10794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Sending DeleteTablet for 3 replicas of tablet 262182f3977c4e0d855b50cc12214eac
I20260925 05:06:59.354277 11210 tablet_service.cc:1514] Processing DeleteTablet for tablet 262182f3977c4e0d855b50cc12214eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:59 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:34096
I20260925 05:06:59.355324 11357 tablet_replica.cc:330] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: stopping tablet replica
I20260925 05:06:59.355320 10931 tablet_service.cc:1514] Processing DeleteTablet for tablet 262182f3977c4e0d855b50cc12214eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:59 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55050
I20260925 05:06:59.355993 11357 raft_consensus.cc:2229] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:59.356439 11357 raft_consensus.cc:2258] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:59.357405 11071 tablet_service.cc:1514] Processing DeleteTablet for tablet 262182f3977c4e0d855b50cc12214eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:59 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:47342
I20260925 05:06:59.358618 11357 ts_tablet_manager.cc:1905] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:59.361189 11359 tablet_replica.cc:330] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: stopping tablet replica
I20260925 05:06:59.362488 10794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:59.363174 10794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:59.365152 11359 raft_consensus.cc:2229] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:59.365562 11359 raft_consensus.cc:2258] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:59.367885 11357 ts_tablet_manager.cc:1918] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:59.368110 11357 log.cc:1196] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/262182f3977c4e0d855b50cc12214eac
I20260925 05:06:59.368848 11357 ts_tablet_manager.cc:1939] T 262182f3977c4e0d855b50cc12214eac P e69f42ffe4db4b2c8530946d3d44e93a: Deleting consensus metadata
I20260925 05:06:59.372260 10930 tablet_service.cc:1467] Processing CreateTablet for tablet a231d22e9a0e4966bb34f31613af4d14 (DEFAULT_TABLE table=test-table [id=48a87271b9024d7ba4a318499a28814b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:59.372460 10781 catalog_manager.cc:4841] TS e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305): tablet 262182f3977c4e0d855b50cc12214eac (table test-table [id=1a0ba417081a4fe48ed70c253ee24e1b]) successfully deleted
I20260925 05:06:59.373061 10930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a231d22e9a0e4966bb34f31613af4d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:59.373032 11210 tablet_service.cc:1467] Processing CreateTablet for tablet a231d22e9a0e4966bb34f31613af4d14 (DEFAULT_TABLE table=test-table [id=48a87271b9024d7ba4a318499a28814b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:59.373416 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a231d22e9a0e4966bb34f31613af4d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:59.377323 11071 tablet_service.cc:1467] Processing CreateTablet for tablet a231d22e9a0e4966bb34f31613af4d14 (DEFAULT_TABLE table=test-table [id=48a87271b9024d7ba4a318499a28814b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:59.377769 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a231d22e9a0e4966bb34f31613af4d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:59.380219 11324 tablet_bootstrap.cc:492] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Bootstrap starting.
I20260925 05:06:59.382023 11324 tablet_bootstrap.cc:654] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:59.388772 11359 ts_tablet_manager.cc:1905] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:59.391523 11324 tablet_bootstrap.cc:492] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: No bootstrap required, opened a new log
I20260925 05:06:59.391830 11324 ts_tablet_manager.cc:1397] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260925 05:06:59.392668 11324 raft_consensus.cc:348] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.393069 11324 raft_consensus.cc:374] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:59.393194 11324 raft_consensus.cc:729] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Initialized, Role: FOLLOWER
I20260925 05:06:59.393527 11324 consensus_queue.cc:260] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.397168 11324 ts_tablet_manager.cc:1428] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 05:06:59.405411 11322 tablet_bootstrap.cc:492] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Bootstrap starting.
I20260925 05:06:59.405212 11359 ts_tablet_manager.cc:1918] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:59.406867 11359 log.cc:1196] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/262182f3977c4e0d855b50cc12214eac
I20260925 05:06:59.407330 11322 tablet_bootstrap.cc:654] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:59.407406 11359 ts_tablet_manager.cc:1939] T 262182f3977c4e0d855b50cc12214eac P b47de446fed44560b9190ecb40169a28: Deleting consensus metadata
I20260925 05:06:59.409373 10780 catalog_manager.cc:4841] TS b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943): tablet 262182f3977c4e0d855b50cc12214eac (table test-table [id=1a0ba417081a4fe48ed70c253ee24e1b]) successfully deleted
I20260925 05:06:59.410248 11323 tablet_bootstrap.cc:492] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Bootstrap starting.
I20260925 05:06:59.412220 11323 tablet_bootstrap.cc:654] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:59.419030 11358 tablet_replica.cc:330] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: stopping tablet replica
I20260925 05:06:59.419499 11358 raft_consensus.cc:2229] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:59.420120 11358 raft_consensus.cc:2258] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:59.422212 11358 ts_tablet_manager.cc:1905] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:59.423244 11323 tablet_bootstrap.cc:492] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: No bootstrap required, opened a new log
I20260925 05:06:59.423477 11323 ts_tablet_manager.cc:1397] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20260925 05:06:59.424217 11322 tablet_bootstrap.cc:492] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: No bootstrap required, opened a new log
I20260925 05:06:59.424414 11322 ts_tablet_manager.cc:1397] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.009s
I20260925 05:06:59.424408 11323 raft_consensus.cc:348] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.425029 11323 raft_consensus.cc:374] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:59.425149 11323 raft_consensus.cc:729] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47de446fed44560b9190ecb40169a28, State: Initialized, Role: FOLLOWER
I20260925 05:06:59.425280 11322 raft_consensus.cc:348] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.425561 11322 raft_consensus.cc:374] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:59.425645 11322 raft_consensus.cc:729] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Initialized, Role: FOLLOWER
I20260925 05:06:59.425493 11323 consensus_queue.cc:260] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.425885 11322 consensus_queue.cc:260] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.427186 11322 ts_tablet_manager.cc:1428] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:59.433487 11323 ts_tablet_manager.cc:1428] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20260925 05:06:59.444113 11358 ts_tablet_manager.cc:1918] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:59.444401 11358 log.cc:1196] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/262182f3977c4e0d855b50cc12214eac
I20260925 05:06:59.445102 11358 ts_tablet_manager.cc:1939] T 262182f3977c4e0d855b50cc12214eac P 673a3092070944a7a155e67df40ebffa: Deleting consensus metadata
I20260925 05:06:59.447108 10778 catalog_manager.cc:4841] TS 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311): tablet 262182f3977c4e0d855b50cc12214eac (table test-table [id=1a0ba417081a4fe48ed70c253ee24e1b]) successfully deleted
I20260925 05:06:59.477466 11331 raft_consensus.cc:482] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:59.477762 11331 raft_consensus.cc:504] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.478740 11331 leader_election.cc:288] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311), b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:59.479378 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a231d22e9a0e4966bb34f31613af4d14" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa" is_pre_election: true
I20260925 05:06:59.479673 10956 raft_consensus.cc:2454] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 0.
I20260925 05:06:59.479714 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a231d22e9a0e4966bb34f31613af4d14" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28" is_pre_election: true
I20260925 05:06:59.480005 11090 raft_consensus.cc:2454] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 0.
I20260925 05:06:59.480232 11161 leader_election.cc:302] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:06:59.480693 11331 raft_consensus.cc:2781] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:59.480832 11331 raft_consensus.cc:482] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:59.480963 11331 raft_consensus.cc:3037] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:59.484344 11331 raft_consensus.cc:504] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.485443 11331 leader_election.cc:288] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 election: Requested vote from peers 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311), b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943)
I20260925 05:06:59.485857 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a231d22e9a0e4966bb34f31613af4d14" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa"
I20260925 05:06:59.485896 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a231d22e9a0e4966bb34f31613af4d14" candidate_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28"
I20260925 05:06:59.486099 10956 raft_consensus.cc:3037] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:59.486152 11090 raft_consensus.cc:3037] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:59.489583 10956 raft_consensus.cc:2454] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 1.
I20260925 05:06:59.490236 11161 leader_election.cc:302] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:06:59.490792 11090 raft_consensus.cc:2454] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69f42ffe4db4b2c8530946d3d44e93a in term 1.
I20260925 05:06:59.491325 11331 raft_consensus.cc:2781] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:59.491619 11331 raft_consensus.cc:686] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 LEADER]: Becoming Leader. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Running, Role: LEADER
I20260925 05:06:59.491997 11331 consensus_queue.cc:237] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:06:59.495270 10794 catalog_manager.cc:5495] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a reported cstate change: term changed from 0 to 1, leader changed from <none> to e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3). New cstate: current_term: 1 leader_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:59.541035 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.533878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.8.215.3:36305)
0925 05:06:59.534056 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.534061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.534080 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:06:59.534144 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.534153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.534241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.534334 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:59.534903 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.536067 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.538146 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.538821 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.538953 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.539749 (+ 796us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.539950 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.540057 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.540592 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.540700 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:06:59.558442 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.551940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:36797 (local address 127.8.215.3:36305)
0925 05:06:59.552106 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:59.552114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:59.552145 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:06:59.552297 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:59.552303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:59.552402 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:59.552500 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:59.552965 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.554178 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.556254 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.557010 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.557144 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.557584 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.557738 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.557830 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:59.557978 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 05:06:59.558062 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:06:59.558476 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:59.551566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.8.215.3:36797)
0925 05:06:59.552009 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:59.552031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:59.552158 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:59.552544 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:59.552549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:59.552562 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:59.552815 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.552828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.554330 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.554335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.556099 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:59.556113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:59.557168 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:59.557175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:59.557358 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:59.557409 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:59.557855 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:59.557868 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:06:59.557957 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:06:59.558115 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260925 05:06:59.569700 11090 raft_consensus.cc:1261] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Refusing update from remote peer e69f42ffe4db4b2c8530946d3d44e93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:59.570284 10956 raft_consensus.cc:1261] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Refusing update from remote peer e69f42ffe4db4b2c8530946d3d44e93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:59.570744 11328 consensus_queue.cc:1035] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:59.571506 11328 consensus_queue.cc:1035] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:06:59.618274 11336 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:07:01.222580 10793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:01.222790 10793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:01.226521 10793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Sending DeleteTablet for 3 replicas of tablet a231d22e9a0e4966bb34f31613af4d14
I20260925 05:07:01.227313 10931 tablet_service.cc:1514] Processing DeleteTablet for tablet a231d22e9a0e4966bb34f31613af4d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55050
I20260925 05:07:01.227511 11210 tablet_service.cc:1514] Processing DeleteTablet for tablet a231d22e9a0e4966bb34f31613af4d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:34096
I20260925 05:07:01.227598 11071 tablet_service.cc:1514] Processing DeleteTablet for tablet a231d22e9a0e4966bb34f31613af4d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:47342
I20260925 05:07:01.243068 11389 tablet_replica.cc:330] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: stopping tablet replica
I20260925 05:07:01.243221 11390 tablet_replica.cc:330] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: stopping tablet replica
I20260925 05:07:01.243350 11388 tablet_replica.cc:330] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: stopping tablet replica
I20260925 05:07:01.243469 11390 raft_consensus.cc:2229] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:01.243474 11389 raft_consensus.cc:2229] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:01.243659 11388 raft_consensus.cc:2229] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:01.243755 11390 raft_consensus.cc:2258] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:01.244159 11389 raft_consensus.cc:2258] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:01.244071 11388 raft_consensus.cc:2258] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:01.245096 11390 ts_tablet_manager.cc:1905] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:01.245460 11388 ts_tablet_manager.cc:1905] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:01.249478 11389 ts_tablet_manager.cc:1905] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:01.250105 11390 ts_tablet_manager.cc:1918] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:07:01.250293 11390 log.cc:1196] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/a231d22e9a0e4966bb34f31613af4d14
I20260925 05:07:01.250809 11390 ts_tablet_manager.cc:1939] T a231d22e9a0e4966bb34f31613af4d14 P b47de446fed44560b9190ecb40169a28: Deleting consensus metadata
I20260925 05:07:01.252031 10780 catalog_manager.cc:4841] TS b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943): tablet a231d22e9a0e4966bb34f31613af4d14 (table test-table [id=48a87271b9024d7ba4a318499a28814b]) successfully deleted
I20260925 05:07:01.253314 11388 ts_tablet_manager.cc:1918] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:07:01.253458 11388 log.cc:1196] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/a231d22e9a0e4966bb34f31613af4d14
I20260925 05:07:01.253938 11388 ts_tablet_manager.cc:1939] T a231d22e9a0e4966bb34f31613af4d14 P 673a3092070944a7a155e67df40ebffa: Deleting consensus metadata
I20260925 05:07:01.254884 10778 catalog_manager.cc:4841] TS 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311): tablet a231d22e9a0e4966bb34f31613af4d14 (table test-table [id=48a87271b9024d7ba4a318499a28814b]) successfully deleted
I20260925 05:07:01.256268 11389 ts_tablet_manager.cc:1918] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:07:01.256415 11389 log.cc:1196] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/a231d22e9a0e4966bb34f31613af4d14
I20260925 05:07:01.256880 11389 ts_tablet_manager.cc:1939] T a231d22e9a0e4966bb34f31613af4d14 P e69f42ffe4db4b2c8530946d3d44e93a: Deleting consensus metadata
I20260925 05:07:01.258002 10781 catalog_manager.cc:4841] TS e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305): tablet a231d22e9a0e4966bb34f31613af4d14 (table test-table [id=48a87271b9024d7ba4a318499a28814b]) successfully deleted
Sep 25 05:07:01 dist-test-slave-wpdm krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:07:01 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:01.290282 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:01.275394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.8.215.62:34773)
0925 05:07:01.275959 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:01.275966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:01.276075 (+ 109us) server_negotiation.cc:407] Connection header received
0925 05:07:01.276301 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:01.276304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:01.276372 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:01.276484 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:01.277865 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.278621 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.279937 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.280319 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.280458 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:01.284288 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:01.284293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:01.284296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:01.284335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:01.287658 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:01.288296 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:01.288308 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:01.288311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:01.288433 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:01.288837 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:01.288844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:01.288849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:01.289121 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:01.289369 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:01.289666 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 05:07:01.289875 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
I20260925 05:07:01.294062 10793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:01.294807 10793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:01.302356 10931 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2c1c07a25f4d0c9cfd470262ec9682 (DEFAULT_TABLE table=test-table [id=17ec75f9cbec44cfafcb12e907fdfa26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:01.302603 11210 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2c1c07a25f4d0c9cfd470262ec9682 (DEFAULT_TABLE table=test-table [id=17ec75f9cbec44cfafcb12e907fdfa26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:01.302953 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2c1c07a25f4d0c9cfd470262ec9682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:01.303026 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2c1c07a25f4d0c9cfd470262ec9682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:01.305841 11071 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2c1c07a25f4d0c9cfd470262ec9682 (DEFAULT_TABLE table=test-table [id=17ec75f9cbec44cfafcb12e907fdfa26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:01.306299 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2c1c07a25f4d0c9cfd470262ec9682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:01.308575 11399 tablet_bootstrap.cc:492] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Bootstrap starting.
I20260925 05:07:01.309950 11399 tablet_bootstrap.cc:654] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:01.312577 11401 tablet_bootstrap.cc:492] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Bootstrap starting.
I20260925 05:07:01.312675 11399 tablet_bootstrap.cc:492] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: No bootstrap required, opened a new log
I20260925 05:07:01.313405 11399 ts_tablet_manager.cc:1397] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:07:01.314350 11399 raft_consensus.cc:348] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.314634 11399 raft_consensus.cc:374] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:01.314738 11399 raft_consensus.cc:729] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Initialized, Role: FOLLOWER
I20260925 05:07:01.315012 11399 consensus_queue.cc:260] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.315647 11399 ts_tablet_manager.cc:1428] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:07:01.316022 11401 tablet_bootstrap.cc:654] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:01.316439 11402 tablet_bootstrap.cc:492] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Bootstrap starting.
I20260925 05:07:01.318331 11402 tablet_bootstrap.cc:654] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:01.319379 11401 tablet_bootstrap.cc:492] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: No bootstrap required, opened a new log
I20260925 05:07:01.319591 11401 ts_tablet_manager.cc:1397] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 05:07:01.320796 11402 tablet_bootstrap.cc:492] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: No bootstrap required, opened a new log
I20260925 05:07:01.320988 11402 ts_tablet_manager.cc:1397] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 05:07:01.320561 11401 raft_consensus.cc:348] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.321168 11401 raft_consensus.cc:374] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:01.321353 11401 raft_consensus.cc:729] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Initialized, Role: FOLLOWER
I20260925 05:07:01.321655 11401 consensus_queue.cc:260] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.322003 11402 raft_consensus.cc:348] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.322347 11402 raft_consensus.cc:374] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:01.322484 11402 raft_consensus.cc:729] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47de446fed44560b9190ecb40169a28, State: Initialized, Role: FOLLOWER
I20260925 05:07:01.322794 11401 ts_tablet_manager.cc:1428] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:01.322795 11402 consensus_queue.cc:260] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.323704 11402 ts_tablet_manager.cc:1428] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:01.584954 11406 raft_consensus.cc:482] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:01.585229 11406 raft_consensus.cc:504] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.586123 11406 leader_election.cc:288] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311), e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305)
I20260925 05:07:01.586788 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2c1c07a25f4d0c9cfd470262ec9682" candidate_uuid: "b47de446fed44560b9190ecb40169a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" is_pre_election: true
I20260925 05:07:01.586787 10955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2c1c07a25f4d0c9cfd470262ec9682" candidate_uuid: "b47de446fed44560b9190ecb40169a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa" is_pre_election: true
I20260925 05:07:01.587050 11235 raft_consensus.cc:2454] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b47de446fed44560b9190ecb40169a28 in term 0.
I20260925 05:07:01.587050 10955 raft_consensus.cc:2454] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b47de446fed44560b9190ecb40169a28 in term 0.
I20260925 05:07:01.587544 11024 leader_election.cc:302] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, b47de446fed44560b9190ecb40169a28, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:07:01.587925 11406 raft_consensus.cc:2781] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:01.588024 11406 raft_consensus.cc:482] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:01.588085 11406 raft_consensus.cc:3037] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:01.641754 11406 raft_consensus.cc:504] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.642520 11406 leader_election.cc:288] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [CANDIDATE]: Term 1 election: Requested vote from peers 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311), e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305)
I20260925 05:07:01.643270 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2c1c07a25f4d0c9cfd470262ec9682" candidate_uuid: "b47de446fed44560b9190ecb40169a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a"
I20260925 05:07:01.643373 10955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2c1c07a25f4d0c9cfd470262ec9682" candidate_uuid: "b47de446fed44560b9190ecb40169a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673a3092070944a7a155e67df40ebffa"
I20260925 05:07:01.643505 11235 raft_consensus.cc:3037] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:01.643620 10955 raft_consensus.cc:3037] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:01.645977 11235 raft_consensus.cc:2454] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47de446fed44560b9190ecb40169a28 in term 1.
I20260925 05:07:01.646479 11024 leader_election.cc:302] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b47de446fed44560b9190ecb40169a28, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:07:01.646791 11406 raft_consensus.cc:2781] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:01.647090 10955 raft_consensus.cc:2454] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47de446fed44560b9190ecb40169a28 in term 1.
I20260925 05:07:01.647511 11406 raft_consensus.cc:686] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 1 LEADER]: Becoming Leader. State: Replica: b47de446fed44560b9190ecb40169a28, State: Running, Role: LEADER
I20260925 05:07:01.648087 11406 consensus_queue.cc:237] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.651599 10792 catalog_manager.cc:5495] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to b47de446fed44560b9190ecb40169a28 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "b47de446fed44560b9190ecb40169a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:01.699025 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:01.693272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.8.215.2:32943)
0925 05:07:01.693866 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:01.693870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:01.693889 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:07:01.693988 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:01.693992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:01.694061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:01.694200 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:01.694557 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.695340 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.696682 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.697169 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.697267 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:01.697677 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:01.698344 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:01.698448 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:01.698594 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:07:01.698658 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":374,"threads_started":1}
I20260925 05:07:01.714813 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:01.709231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:36305 (local address 127.8.215.2:34391)
0925 05:07:01.709913 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:01.709929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:01.710086 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:01.710375 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:01.710379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:01.710388 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:07:01.710672 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:01.710679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.711724 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.711728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:01.712950 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:01.712962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.713603 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.713607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:01.713735 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:01.713790 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:07:01.714279 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:07:01.714288 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:07:01.714354 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 05:07:01.714509 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":371,"threads_started":1}
I20260925 05:07:01.714805 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:01.709395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:34391 (local address 127.8.215.3:36305)
0925 05:07:01.709937 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:01.709942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:01.710040 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:07:01.710194 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:01.710197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:01.710261 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:01.710345 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:01.710836 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.711592 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.713095 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.713476 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.713569 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:01.714032 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:01.714192 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:01.714270 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:01.714386 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 05:07:01.714473 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":296,"threads_started":1}
I20260925 05:07:01.723953 11235 raft_consensus.cc:1261] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Refusing update from remote peer b47de446fed44560b9190ecb40169a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:01.724076 10955 raft_consensus.cc:1261] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Refusing update from remote peer b47de446fed44560b9190ecb40169a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:01.724800 11407 consensus_queue.cc:1035] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:01.725154 11406 consensus_queue.cc:1035] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:01.746608 11372 mvcc.cc:205] Tried to move back new op lower bound from 7333121317774909440 to 7333121317474553856. Current Snapshot: MvccSnapshot[applied={T|T < 7333121317774909440}]
I20260925 05:07:01.763206 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:01.758054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:36629 (local address 127.8.215.2:32943)
0925 05:07:01.758487 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:01.758493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:01.758668 (+ 175us) server_negotiation.cc:407] Connection header received
0925 05:07:01.758822 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:01.758825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:01.758886 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:01.758959 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:01.759338 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.760081 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.761692 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.762088 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.762235 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:01.762696 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:01.762791 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:01.762859 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:01.762950 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:07:01.763008 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260925 05:07:01.763329 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:01.757926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.8.215.3:36629)
0925 05:07:01.758594 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:01.758610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:01.758734 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:01.758986 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:01.758989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:01.758998 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:07:01.759197 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:01.759203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.760197 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.760200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:01.761559 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:01.761571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:01.762316 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:01.762325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:01.762495 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:01.762545 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:07:01.762874 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:07:01.762882 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:07:01.762937 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:07:01.763035 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":407,"threads_started":1}
I20260925 05:07:01.795229 10794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:01.795535 10794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:01.798854 10794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Sending DeleteTablet for 3 replicas of tablet 6a2c1c07a25f4d0c9cfd470262ec9682
I20260925 05:07:01.799662 10931 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2c1c07a25f4d0c9cfd470262ec9682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55050
I20260925 05:07:01.800036 11071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2c1c07a25f4d0c9cfd470262ec9682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:47342
I20260925 05:07:01.800477 11422 tablet_replica.cc:330] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: stopping tablet replica
I20260925 05:07:01.800817 11422 raft_consensus.cc:2229] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:01.801148 11422 raft_consensus.cc:2258] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:01.801280 11423 tablet_replica.cc:330] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: stopping tablet replica
I20260925 05:07:01.801321 11210 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2c1c07a25f4d0c9cfd470262ec9682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:34096
I20260925 05:07:01.801471 11423 raft_consensus.cc:2229] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:01.802066 11423 raft_consensus.cc:2258] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:01.802438 10794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:07:01.802637 11422 ts_tablet_manager.cc:1905] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 05:07:01.803208 10794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:01.803319 11423 ts_tablet_manager.cc:1905] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:01.808422 11424 tablet_replica.cc:330] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: stopping tablet replica
I20260925 05:07:01.808633 11424 raft_consensus.cc:2229] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:01.808944 11424 raft_consensus.cc:2258] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:01.810151 11424 ts_tablet_manager.cc:1905] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:01.810949 10931 tablet_service.cc:1467] Processing CreateTablet for tablet 6439d6a8eb9b4d8bae3db58ae504565b (DEFAULT_TABLE table=test-table [id=a5ff1dcef7c34e4b88454763ba5b462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:01.811793 11209 tablet_service.cc:1467] Processing CreateTablet for tablet 6439d6a8eb9b4d8bae3db58ae504565b (DEFAULT_TABLE table=test-table [id=a5ff1dcef7c34e4b88454763ba5b462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:01.812318 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6439d6a8eb9b4d8bae3db58ae504565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:01.812561 11071 tablet_service.cc:1467] Processing CreateTablet for tablet 6439d6a8eb9b4d8bae3db58ae504565b (DEFAULT_TABLE table=test-table [id=a5ff1dcef7c34e4b88454763ba5b462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:01.851631 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6439d6a8eb9b4d8bae3db58ae504565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:01.851630 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6439d6a8eb9b4d8bae3db58ae504565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:01.864131 11425 tablet_bootstrap.cc:492] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: Bootstrap starting.
I20260925 05:07:01.866137 11425 tablet_bootstrap.cc:654] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:01.871671 11426 tablet_bootstrap.cc:492] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Bootstrap starting.
I20260925 05:07:01.874389 11426 tablet_bootstrap.cc:654] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:01.875946 11428 tablet_bootstrap.cc:492] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Bootstrap starting.
I20260925 05:07:01.877807 11426 tablet_bootstrap.cc:492] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: No bootstrap required, opened a new log
I20260925 05:07:01.877892 11428 tablet_bootstrap.cc:654] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:01.877997 11426 ts_tablet_manager.cc:1397] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 05:07:01.878895 11426 raft_consensus.cc:348] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.879175 11426 raft_consensus.cc:374] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:01.879343 11426 raft_consensus.cc:729] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47de446fed44560b9190ecb40169a28, State: Initialized, Role: FOLLOWER
I20260925 05:07:01.879701 11426 consensus_queue.cc:260] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.881028 11425 tablet_bootstrap.cc:492] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: No bootstrap required, opened a new log
I20260925 05:07:01.881281 11425 ts_tablet_manager.cc:1397] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260925 05:07:01.881300 11428 tablet_bootstrap.cc:492] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: No bootstrap required, opened a new log
I20260925 05:07:01.881510 11428 ts_tablet_manager.cc:1397] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 05:07:01.882332 11428 raft_consensus.cc:348] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.882573 11428 raft_consensus.cc:374] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:01.882686 11428 raft_consensus.cc:729] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69f42ffe4db4b2c8530946d3d44e93a, State: Initialized, Role: FOLLOWER
I20260925 05:07:01.882406 11425 raft_consensus.cc:348] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.882934 11425 raft_consensus.cc:374] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:01.882963 11426 ts_tablet_manager.cc:1428] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:07:01.882941 11428 consensus_queue.cc:260] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.883082 11425 raft_consensus.cc:729] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Initialized, Role: FOLLOWER
I20260925 05:07:01.883397 11425 consensus_queue.cc:260] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.884054 11425 ts_tablet_manager.cc:1428] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:01.884243 11428 ts_tablet_manager.cc:1428] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:01.951857 11423 ts_tablet_manager.cc:1918] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:01.951869 11422 ts_tablet_manager.cc:1918] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:01.952044 11422 log.cc:1196] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/6a2c1c07a25f4d0c9cfd470262ec9682
I20260925 05:07:01.952060 11423 log.cc:1196] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/6a2c1c07a25f4d0c9cfd470262ec9682
I20260925 05:07:01.952531 11422 ts_tablet_manager.cc:1939] T 6a2c1c07a25f4d0c9cfd470262ec9682 P 673a3092070944a7a155e67df40ebffa: Deleting consensus metadata
I20260925 05:07:01.952533 11423 ts_tablet_manager.cc:1939] T 6a2c1c07a25f4d0c9cfd470262ec9682 P b47de446fed44560b9190ecb40169a28: Deleting consensus metadata
I20260925 05:07:01.953107 11424 ts_tablet_manager.cc:1918] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:01.953248 11424 log.cc:1196] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/6a2c1c07a25f4d0c9cfd470262ec9682
I20260925 05:07:01.953709 11424 ts_tablet_manager.cc:1939] T 6a2c1c07a25f4d0c9cfd470262ec9682 P e69f42ffe4db4b2c8530946d3d44e93a: Deleting consensus metadata
I20260925 05:07:01.953976 10780 catalog_manager.cc:4841] TS b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943): tablet 6a2c1c07a25f4d0c9cfd470262ec9682 (table test-table [id=17ec75f9cbec44cfafcb12e907fdfa26]) successfully deleted
I20260925 05:07:01.954867 10781 catalog_manager.cc:4841] TS e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305): tablet 6a2c1c07a25f4d0c9cfd470262ec9682 (table test-table [id=17ec75f9cbec44cfafcb12e907fdfa26]) successfully deleted
I20260925 05:07:01.955147 10778 catalog_manager.cc:4841] TS 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311): tablet 6a2c1c07a25f4d0c9cfd470262ec9682 (table test-table [id=17ec75f9cbec44cfafcb12e907fdfa26]) successfully deleted
I20260925 05:07:01.966423 11405 raft_consensus.cc:482] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:01.966679 11405 raft_consensus.cc:504] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:01.967540 11405 leader_election.cc:288] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943), e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305)
I20260925 05:07:01.968168 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6439d6a8eb9b4d8bae3db58ae504565b" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28" is_pre_election: true
I20260925 05:07:01.968211 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6439d6a8eb9b4d8bae3db58ae504565b" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" is_pre_election: true
I20260925 05:07:01.968444 11235 raft_consensus.cc:2454] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 0.
I20260925 05:07:01.968449 11090 raft_consensus.cc:2454] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 0.
I20260925 05:07:01.968935 10884 leader_election.cc:302] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:07:01.969352 11405 raft_consensus.cc:2781] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:01.969476 11405 raft_consensus.cc:482] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:01.969591 11405 raft_consensus.cc:3037] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:02.057991 11405 raft_consensus.cc:504] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:02.058786 11405 leader_election.cc:288] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 election: Requested vote from peers b47de446fed44560b9190ecb40169a28 (127.8.215.2:32943), e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305)
I20260925 05:07:02.059415 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6439d6a8eb9b4d8bae3db58ae504565b" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47de446fed44560b9190ecb40169a28"
I20260925 05:07:02.059545 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6439d6a8eb9b4d8bae3db58ae504565b" candidate_uuid: "673a3092070944a7a155e67df40ebffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69f42ffe4db4b2c8530946d3d44e93a"
I20260925 05:07:02.059689 11090 raft_consensus.cc:3037] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:02.059742 11235 raft_consensus.cc:3037] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:02.062924 11235 raft_consensus.cc:2454] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 1.
I20260925 05:07:02.062961 11090 raft_consensus.cc:2454] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673a3092070944a7a155e67df40ebffa in term 1.
I20260925 05:07:02.063457 10884 leader_election.cc:302] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673a3092070944a7a155e67df40ebffa, e69f42ffe4db4b2c8530946d3d44e93a; no voters:
I20260925 05:07:02.063977 11405 raft_consensus.cc:2781] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:02.064493 11405 raft_consensus.cc:686] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 1 LEADER]: Becoming Leader. State: Replica: 673a3092070944a7a155e67df40ebffa, State: Running, Role: LEADER
I20260925 05:07:02.064896 11405 consensus_queue.cc:237] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } }
I20260925 05:07:02.067888 10794 catalog_manager.cc:5495] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 673a3092070944a7a155e67df40ebffa (127.8.215.1). New cstate: current_term: 1 leader_uuid: "673a3092070944a7a155e67df40ebffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "673a3092070944a7a155e67df40ebffa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 42311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:02.141765 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:02.135526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.8.215.1:42311)
0925 05:07:02.136342 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:02.136346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:02.136366 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:07:02.136417 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:02.136421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:02.136479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:02.136556 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:02.137095 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:02.137850 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:02.139804 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:02.140256 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:02.140374 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:02.140951 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:02.141061 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:02.141130 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:02.141318 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 05:07:02.141391 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":527,"threads_started":1}
I20260925 05:07:02.154342 11235 raft_consensus.cc:1261] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Refusing update from remote peer 673a3092070944a7a155e67df40ebffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:02.154604 11090 raft_consensus.cc:1261] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Refusing update from remote peer 673a3092070944a7a155e67df40ebffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:02.155061 11431 consensus_queue.cc:1035] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69f42ffe4db4b2c8530946d3d44e93a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:02.155419 11405 consensus_queue.cc:1035] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b47de446fed44560b9190ecb40169a28" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:07:02.189216 11372 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:07:03.780458 10792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:03.780793 10792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:03.785107 10792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d6ae535aa464ec0910a06a81ecec850: Sending DeleteTablet for 3 replicas of tablet 6439d6a8eb9b4d8bae3db58ae504565b
I20260925 05:07:03.785925 10931 tablet_service.cc:1514] Processing DeleteTablet for tablet 6439d6a8eb9b4d8bae3db58ae504565b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:03 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55050
I20260925 05:07:03.786028 11071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6439d6a8eb9b4d8bae3db58ae504565b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:03 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:47342
I20260925 05:07:03.786314 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 6439d6a8eb9b4d8bae3db58ae504565b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:03 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:34096
I20260925 05:07:03.786938 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10865
I20260925 05:07:03.791309 11463 tablet_replica.cc:330] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: stopping tablet replica
I20260925 05:07:03.791455 11462 tablet_replica.cc:330] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: stopping tablet replica
I20260925 05:07:03.791620 11463 raft_consensus.cc:2229] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:03.791857 11462 raft_consensus.cc:2229] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:03.792029 11463 raft_consensus.cc:2258] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:03.792219 11462 raft_consensus.cc:2258] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:03.793494 11462 ts_tablet_manager.cc:1905] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:03.793581 11463 ts_tablet_manager.cc:1905] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:03.797411 11461 tablet_replica.cc:330] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: stopping tablet replica
I20260925 05:07:03.797636 11461 raft_consensus.cc:2229] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:03.798022 11461 raft_consensus.cc:2258] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:03.799000 11461 ts_tablet_manager.cc:1905] T 6439d6a8eb9b4d8bae3db58ae504565b P 673a3092070944a7a155e67df40ebffa: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 05:07:03 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:03.816864 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.791583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.8.215.1:42311)
0925 05:07:03.797645 (+ 6062us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:03.797649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:03.797670 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:07:03.797751 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:03.797758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:03.797839 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:03.797933 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:03.799798 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:03.800699 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:03.802571 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:03.803000 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:03.803095 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:03.811148 (+ 8053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:03.811164 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:03.811181 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:03.811220 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:03.814246 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:03.814940 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:03.814947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:03.814949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:03.815077 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:03.815447 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:03.815450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:03.815452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:03.815882 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:03.816085 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:03.816351 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 05:07:03.816516 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5940,"thread_start_us":5787,"threads_started":1}
W20260925 05:07:03.817523 10996 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38422
W20260925 05:07:03.818177 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:07:03.818404 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 10865
I20260925 05:07:03.840276 11462 ts_tablet_manager.cc:1918] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:07:03.840277 11463 ts_tablet_manager.cc:1918] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:07:03.840431 11463 log.cc:1196] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/6439d6a8eb9b4d8bae3db58ae504565b
I20260925 05:07:03.840597 11462 log.cc:1196] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/6439d6a8eb9b4d8bae3db58ae504565b
I20260925 05:07:03.840972 11463 ts_tablet_manager.cc:1939] T 6439d6a8eb9b4d8bae3db58ae504565b P e69f42ffe4db4b2c8530946d3d44e93a: Deleting consensus metadata
I20260925 05:07:03.841142 11462 ts_tablet_manager.cc:1939] T 6439d6a8eb9b4d8bae3db58ae504565b P b47de446fed44560b9190ecb40169a28: Deleting consensus metadata
I20260925 05:07:03.866699 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11005
W20260925 05:07:03.866796 10778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.215.1:42311 (error 108)
I20260925 05:07:03.869122 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.868196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.0.0.1:38438)
0925 05:07:03.868715 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.868901 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
W20260925 05:07:03.869336 10778 catalog_manager.cc:4584] TS 673a3092070944a7a155e67df40ebffa (127.8.215.1:42311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6439d6a8eb9b4d8bae3db58ae504565b: Network error: Client connection negotiation failed: client connection to 127.8.215.1:42311: connect: Connection refused (error 111)
Sep 25 05:07:03 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:07:03.882519 11467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.867622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.8.215.2:32943)
0925 05:07:03.868156 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:03.868162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:03.868180 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:07:03.868244 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:03.868248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:03.868304 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:03.868419 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:03.869919 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:03.870689 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:03.871911 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:03.872319 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:03.872451 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:03.876566 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:03.876580 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:03.876596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:03.876629 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:03.879648 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:03.880354 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:03.880361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:03.880363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:03.880498 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:03.880930 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:03.880936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:03.880939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:03.881307 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:03.881654 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:03.881955 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 05:07:03.882174 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":264,"threads_started":1}
W20260925 05:07:03.883219 11136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344
W20260925 05:07:03.883833 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11005: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:07:03.883960 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 11005
I20260925 05:07:03.919835 10781 catalog_manager.cc:4841] TS e69f42ffe4db4b2c8530946d3d44e93a (127.8.215.3:36305): tablet 6439d6a8eb9b4d8bae3db58ae504565b (table test-table [id=a5ff1dcef7c34e4b88454763ba5b462b]) successfully deleted
I20260925 05:07:03.923051 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.922419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.0.0.1:38440)
0925 05:07:03.922675 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.922819 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 05:07:03.924288 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.923784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.0.0.1:38454)
0925 05:07:03.923979 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.924097 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 05:07:03.941648 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11145
I20260925 05:07:03.942534 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.942018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.0.0.1:54348)
0925 05:07:03.942226 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.942343 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.2:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
Sep 25 05:07:03 dist-test-slave-wpdm krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:07:03.958526 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.942490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.8.215.3:36305)
0925 05:07:03.943203 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:03.943208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:03.943222 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:07:03.943277 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:03.943281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:03.943340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:03.943464 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:03.944839 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:03.945566 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:03.946832 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:03.947271 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:03.947411 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:03.951973 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:03.951989 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:03.952006 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:03.952041 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:03.955679 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:03.956433 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:03.956441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:03.956443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:03.956559 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:03.956964 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:03.956971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:03.956974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:03.957324 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:03.957674 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:03.957951 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 05:07:03.958183 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":456,"threads_started":1}
W20260925 05:07:03.959263 11275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128
W20260925 05:07:03.959949 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11145: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:07:03.960095 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 11145
I20260925 05:07:03.975101 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.974563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.0.0.1:54358)
0925 05:07:03.974785 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.974899 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.2:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 05:07:03.976145 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.975792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.0.0.1:54362)
0925 05:07:03.975967 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.976036 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.2:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:07:03.985610 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.985134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.0.0.1:38460)
0925 05:07:03.985347 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.985438 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 05:07:03.986972 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:03.986441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:42311 (local address 127.0.0.1:38476)
0925 05:07:03.986658 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:03.986772 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:07:03.987309 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 10762
I20260925 05:07:03.988395 10834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:04.112679 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:04.112105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:32943 (local address 127.0.0.1:54364)
0925 05:07:04.112350 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:04.112462 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.2:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 05:07:04.112901 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 10762
2026-09-25T05:07:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9184 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:06 dist-test-slave-wpdm krb5kdc[11474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:06 dist-test-slave-wpdm krb5kdc[11474](info): set up 2 sockets
Sep 25 05:07:06 dist-test-slave-wpdm krb5kdc[11474](info): commencing operation
krb5kdc: starting...
W20260925 05:07:07.362074 9052 mini_kdc.cc:121] Time spent starting KDC: real 3.207s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:07 dist-test-slave-wpdm krb5kdc[11474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:07.573591 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35413
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:07.916885 11487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:07.917214 11487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:07.917412 11487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:07.924952 11487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:07.925125 11487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:07.925237 11487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:07.925338 11487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:07.925426 11487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:07.935983 11487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:35413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:07.937570 11487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:07.939028 11487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:07.952418 11495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:07.952490 11493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:07.952507 11492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:07.958386 11487 server_base.cc:1034] running on GCE node
I20260925 05:07:07.959313 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:07.960915 11487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:07.962180 11487 hybrid_clock.cc:648] HybridClock initialized: now 1790312827962109 us; error 62 us; skew 500 ppm
I20260925 05:07:07.962589 11487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:07 dist-test-slave-wpdm krb5kdc[11474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:07.965785 11487 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:07.967480 11487 webserver.cc:460] Webserver started at http://127.8.215.62:34647/ using document root <none> and password file <none>
I20260925 05:07:07.968252 11487 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:07.968403 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:07.968768 11487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:07.971854 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "3af249cb7e3d42ee8404a8d5741f25b2"
format_stamp: "Formatted at 2026-09-25 05:07:07 on dist-test-slave-wpdm"
server_key: "3c68544b2a718ebd3d21c4ba38c13475"
server_key_iv: "c1ffa6a4ee97fc06933e94f05cb010d4"
server_key_version: "encryptionkey@0"
I20260925 05:07:07.972679 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "3af249cb7e3d42ee8404a8d5741f25b2"
format_stamp: "Formatted at 2026-09-25 05:07:07 on dist-test-slave-wpdm"
server_key: "3c68544b2a718ebd3d21c4ba38c13475"
server_key_iv: "c1ffa6a4ee97fc06933e94f05cb010d4"
server_key_version: "encryptionkey@0"
I20260925 05:07:07.978293 11487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:07:07.982100 11502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:07.983605 11487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:07.983807 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "3af249cb7e3d42ee8404a8d5741f25b2"
format_stamp: "Formatted at 2026-09-25 05:07:07 on dist-test-slave-wpdm"
server_key: "3c68544b2a718ebd3d21c4ba38c13475"
server_key_iv: "c1ffa6a4ee97fc06933e94f05cb010d4"
server_key_version: "encryptionkey@0"
I20260925 05:07:07.984020 11487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:08.002315 11487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:08.006649 11487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:08.006960 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:08.026266 11487 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:35413
I20260925 05:07:08.026285 11553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:35413 every 8 connection(s)
I20260925 05:07:08.027763 11487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:08.031733 11554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:08.034216 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 11487
I20260925 05:07:08.034449 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:08.035248 9052 external_mini_cluster.cc:1428] Setting key 16427e61005ba497170bee9012eb1e5f
I20260925 05:07:08.044139 11554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2: Bootstrap starting.
I20260925 05:07:08.047567 11554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:08.048631 11554 log.cc:824] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:07:08 dist-test-slave-wpdm krb5kdc[11474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:08.052062 11554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2: No bootstrap required, opened a new log
I20260925 05:07:08.058542 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.037633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.8.215.62:35413)
0925 05:07:08.038490 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.038503 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.038547 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:07:08.040223 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.040247 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.040657 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.041156 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:08.042643 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.044009 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.045460 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.046200 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.046423 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.050832 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:08.050841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:08.050857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:08.050896 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:08.054131 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.054945 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.054952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.054960 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.055136 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.055535 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.055539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.055541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.056141 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:08.056467 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.056960 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.057398 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":381,"threads_started":1}
I20260925 05:07:08.060344 11554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35413 } }
I20260925 05:07:08.060767 11554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:08.060897 11554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af249cb7e3d42ee8404a8d5741f25b2, State: Initialized, Role: FOLLOWER
I20260925 05:07:08.061525 11554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35413 } }
I20260925 05:07:08.061787 11554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:08.061939 11554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:08.062107 11554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:08.064247 11554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35413 } }
I20260925 05:07:08.064806 11554 leader_election.cc:302] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af249cb7e3d42ee8404a8d5741f25b2; no voters:
I20260925 05:07:08.065412 11554 leader_election.cc:288] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:08.065546 11559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:08.066809 11559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3af249cb7e3d42ee8404a8d5741f25b2, State: Running, Role: LEADER
I20260925 05:07:08.067489 11559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35413 } }
I20260925 05:07:08.067772 11554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:08.070520 11560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35413 } } }
I20260925 05:07:08.070966 11560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:08.071367 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af249cb7e3d42ee8404a8d5741f25b2. Latest consensus state: current_term: 1 leader_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af249cb7e3d42ee8404a8d5741f25b2" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35413 } } }
I20260925 05:07:08.071689 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:08.073550 11567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:08.078732 11567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:08.088631 11567 catalog_manager.cc:1347] Generated new cluster ID: 46594e9e91e54ffeba59678c62a87509
I20260925 05:07:08.088797 11567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:08.112381 11567 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:08.113894 11567 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:08.122957 11567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af249cb7e3d42ee8404a8d5741f25b2: Generated new TSK 0
I20260925 05:07:08.123883 11567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:07:08.182191 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35413
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:08.542047 11580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:08.542351 11580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:08.542538 11580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:08.550303 11580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:08.550437 11580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:08.550611 11580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:08.561539 11580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:08.562857 11580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:08.564203 11580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:08.578150 11585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.578073 11588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.578107 11586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:08.582568 11580 server_base.cc:1034] running on GCE node
I20260925 05:07:08.583563 11580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:08.585052 11580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:08.586314 11580 hybrid_clock.cc:648] HybridClock initialized: now 1790312828586191 us; error 111 us; skew 500 ppm
I20260925 05:07:08.586764 11580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:08 dist-test-slave-wpdm krb5kdc[11474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:08.590750 11580 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:07:08.592334 11580 webserver.cc:460] Webserver started at http://127.8.215.1:43533/ using document root <none> and password file <none>
I20260925 05:07:08.593217 11580 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:08.593357 11580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:08.593677 11580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:08.596942 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "f0f491434ab44ed7b02651f15fbd9293"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-wpdm"
server_key: "92e3c19775fc5f52c9eb9c228db4ffc1"
server_key_iv: "67f22c3b06a210cd50c5bd2355aab157"
server_key_version: "encryptionkey@0"
I20260925 05:07:08.597821 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "f0f491434ab44ed7b02651f15fbd9293"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-wpdm"
server_key: "92e3c19775fc5f52c9eb9c228db4ffc1"
server_key_iv: "67f22c3b06a210cd50c5bd2355aab157"
server_key_version: "encryptionkey@0"
I20260925 05:07:08.603601 11580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:08.607491 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.609042 11580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 05:07:08.609233 11580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "f0f491434ab44ed7b02651f15fbd9293"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-wpdm"
server_key: "92e3c19775fc5f52c9eb9c228db4ffc1"
server_key_iv: "67f22c3b06a210cd50c5bd2355aab157"
server_key_version: "encryptionkey@0"
I20260925 05:07:08.609449 11580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:08.664297 11580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:08.668941 11580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:08.669241 11580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:08.670181 11580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:08.671495 11580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:08.671569 11580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.671657 11580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:08.671701 11580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.704648 11580 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:36553
I20260925 05:07:08.704697 11707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:36553 every 8 connection(s)
I20260925 05:07:08.706027 11580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:08.715345 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 11580
I20260925 05:07:08.715571 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:08.716149 9052 external_mini_cluster.cc:1428] Setting key b8c9ebbd5fd67578e3c1b608a79ed5eb
Sep 25 05:07:08 dist-test-slave-wpdm krb5kdc[11474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:08.726826 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.708576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:58283 (local address 127.8.215.62:35413)
0925 05:07:08.709037 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.709041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.710003 (+ 962us) server_negotiation.cc:407] Connection header received
0925 05:07:08.711906 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.711913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.712006 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.712121 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:08.713767 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.714574 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.716230 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.716638 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.716795 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.720666 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:08.720671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:08.720675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:08.720706 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:08.723240 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.723991 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.724001 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.724004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.724119 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.724558 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.724563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.724566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.724886 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:08.725034 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.726072 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.726341 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":266,"threads_started":1}
I20260925 05:07:08.727569 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.709153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35413 (local address 127.8.215.1:58283)
0925 05:07:08.709838 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.709886 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.711625 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.712360 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.712369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.712964 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:08.713600 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.713621 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.714714 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.714723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.716087 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.716098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.716882 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.716890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.717152 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.717892 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:08.717915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:08.720443 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:08.723396 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.723405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.723425 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.723870 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.724231 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.724235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.724239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.724443 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.725010 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:08.725018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:08.725540 (+ 522us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.725920 (+ 380us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.726433 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":278,"threads_started":1}
I20260925 05:07:08.729239 11708 heartbeater.cc:347] Connected to a master server at 127.8.215.62:35413
I20260925 05:07:08.729561 11708 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:08.730315 11708 heartbeater.cc:511] Master 127.8.215.62:35413 requested a full tablet report, sending...
I20260925 05:07:08.732417 11519 ts_manager.cc:194] Registered new tserver with Master: f0f491434ab44ed7b02651f15fbd9293 (127.8.215.1:36553)
I20260925 05:07:08.733943 11519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:58283
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:07:08.759590 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35413
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:09.091790 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:09.092096 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:09.092186 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:09.099452 11714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:09.099578 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:09.099696 11714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:09.109414 11714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:09.110694 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:09.112033 11714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:09.125231 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:09.125314 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:09.125301 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:09.126178 11714 server_base.cc:1034] running on GCE node
I20260925 05:07:09.126813 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:09.127797 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:09.128983 11714 hybrid_clock.cc:648] HybridClock initialized: now 1790312829128919 us; error 56 us; skew 500 ppm
I20260925 05:07:09.129300 11714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:09 dist-test-slave-wpdm krb5kdc[11474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:09.137095 11714 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:07:09.138706 11714 webserver.cc:460] Webserver started at http://127.8.215.2:43115/ using document root <none> and password file <none>
I20260925 05:07:09.139469 11714 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:09.139614 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:09.139946 11714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:09.143029 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "f2ce3aae82f84d94aad37addc2508cb5"
format_stamp: "Formatted at 2026-09-25 05:07:09 on dist-test-slave-wpdm"
server_key: "455adf502a16502a9a7b0eae06cf02cd"
server_key_iv: "c709d1e01d11e7004754704a288c7eea"
server_key_version: "encryptionkey@0"
I20260925 05:07:09.143882 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "f2ce3aae82f84d94aad37addc2508cb5"
format_stamp: "Formatted at 2026-09-25 05:07:09 on dist-test-slave-wpdm"
server_key: "455adf502a16502a9a7b0eae06cf02cd"
server_key_iv: "c709d1e01d11e7004754704a288c7eea"
server_key_version: "encryptionkey@0"
I20260925 05:07:09.149348 11714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:07:09.152966 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:09.154451 11714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:09.154621 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "f2ce3aae82f84d94aad37addc2508cb5"
format_stamp: "Formatted at 2026-09-25 05:07:09 on dist-test-slave-wpdm"
server_key: "455adf502a16502a9a7b0eae06cf02cd"
server_key_iv: "c709d1e01d11e7004754704a288c7eea"
server_key_version: "encryptionkey@0"
I20260925 05:07:09.154827 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:09.177899 11714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:09.182058 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:09.182361 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:09.183362 11714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:09.184717 11714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:09.184863 11714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:09.185007 11714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:09.185074 11714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:09.218562 11714 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:39905
I20260925 05:07:09.218609 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:39905 every 8 connection(s)
I20260925 05:07:09.219919 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:09.228680 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 11714
I20260925 05:07:09.228910 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:09.229372 9052 external_mini_cluster.cc:1428] Setting key 6f70f57a003c7a00b05124842ce528e7
Sep 25 05:07:09 dist-test-slave-wpdm krb5kdc[11474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:09.241472 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.222668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:58769 (local address 127.8.215.62:35413)
0925 05:07:09.222883 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:09.222889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:09.224088 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 05:07:09.225978 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:09.225982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:09.226053 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:09.226147 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:09.227908 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.228837 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.230674 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.231251 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.231389 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.235659 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:09.235664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:09.235668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:09.235701 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:09.238389 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:09.239156 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:09.239162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:09.239166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:09.239302 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:09.239717 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:09.239724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:09.239727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:09.239988 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:09.240145 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:09.240895 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 05:07:09.241120 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:07:09.242518 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.223201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35413 (local address 127.8.215.2:58769)
0925 05:07:09.223929 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:09.223966 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:09.225699 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:09.226441 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:09.226450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:09.227021 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:09.227736 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.227756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.229022 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.229028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.230531 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.230543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.231503 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.231511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.231806 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.232586 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:09.232621 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:09.235410 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:09.238545 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:09.238555 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:09.238576 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:09.239030 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:09.239414 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:09.239418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:09.239422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:09.239611 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:09.240125 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:09.240133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:09.240481 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 05:07:09.240799 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 05:07:09.241360 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":287,"threads_started":1}
I20260925 05:07:09.244302 11842 heartbeater.cc:347] Connected to a master server at 127.8.215.62:35413
I20260925 05:07:09.244658 11842 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:09.245419 11842 heartbeater.cc:511] Master 127.8.215.62:35413 requested a full tablet report, sending...
I20260925 05:07:09.247088 11519 ts_manager.cc:194] Registered new tserver with Master: f2ce3aae82f84d94aad37addc2508cb5 (127.8.215.2:39905)
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
I20260925 05:07:09.248389 11519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:58769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:07:09.275692 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35413
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:09.609409 11847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:09.609772 11847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:09.609948 11847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:09.617614 11847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:09.617782 11847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:09.617964 11847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:09.627883 11847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:09.629269 11847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:09.630574 11847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:09.644078 11852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:09.643996 11853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:09.644078 11855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:09.645107 11847 server_base.cc:1034] running on GCE node
I20260925 05:07:09.645778 11847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:09.646767 11847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:09.648093 11847 hybrid_clock.cc:648] HybridClock initialized: now 1790312829648011 us; error 41 us; skew 500 ppm
I20260925 05:07:09.648509 11847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:09 dist-test-slave-wpdm krb5kdc[11474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:09.651821 11847 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:07:09.653605 11847 webserver.cc:460] Webserver started at http://127.8.215.3:33505/ using document root <none> and password file <none>
I20260925 05:07:09.654412 11847 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:09.654548 11847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:09.654865 11847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:09.657931 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "84bd6a28486d4fc08ae2a4d185b610d9"
format_stamp: "Formatted at 2026-09-25 05:07:09 on dist-test-slave-wpdm"
server_key: "8fe175386f7d4c308a46e2fb4a7684b3"
server_key_iv: "1eac6dad33274edd60e844ae84747f80"
server_key_version: "encryptionkey@0"
I20260925 05:07:09.658754 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "84bd6a28486d4fc08ae2a4d185b610d9"
format_stamp: "Formatted at 2026-09-25 05:07:09 on dist-test-slave-wpdm"
server_key: "8fe175386f7d4c308a46e2fb4a7684b3"
server_key_iv: "1eac6dad33274edd60e844ae84747f80"
server_key_version: "encryptionkey@0"
I20260925 05:07:09.664186 11847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:07:09.667860 11862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:09.669401 11847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:07:09.669584 11847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "84bd6a28486d4fc08ae2a4d185b610d9"
format_stamp: "Formatted at 2026-09-25 05:07:09 on dist-test-slave-wpdm"
server_key: "8fe175386f7d4c308a46e2fb4a7684b3"
server_key_iv: "1eac6dad33274edd60e844ae84747f80"
server_key_version: "encryptionkey@0"
I20260925 05:07:09.669790 11847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:09.707561 11847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:09.711915 11847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:09.712221 11847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:09.713331 11847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:09.714736 11847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:09.714835 11847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:09.714985 11847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:09.715054 11847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:09.737277 11708 heartbeater.cc:503] Master 127.8.215.62:35413 was elected leader, sending a full tablet report...
I20260925 05:07:09.748353 11847 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:43917
I20260925 05:07:09.748394 11974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:43917 every 8 connection(s)
I20260925 05:07:09.749863 11847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:09.756002 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 11847
I20260925 05:07:09.756213 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:09.756750 9052 external_mini_cluster.cc:1428] Setting key a5cb5f124557661aa06cc8d1605cae99
Sep 25 05:07:09 dist-test-slave-wpdm krb5kdc[11474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:09.769770 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.752348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:58493 (local address 127.8.215.62:35413)
0925 05:07:09.752842 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:09.752846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:09.753927 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 05:07:09.755920 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:09.755924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:09.755992 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:09.756135 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:09.757969 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.759059 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.760817 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.761209 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.761311 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.764605 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:09.764610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:09.764613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:09.764643 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:09.766990 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:09.767641 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:09.767645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:09.767647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:09.767760 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:09.768176 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:09.768179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:09.768181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:09.768438 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:09.768542 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:09.769284 (+ 742us) server_negotiation.cc:299] Negotiation successful
0925 05:07:09.769456 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":277,"threads_started":1}
I20260925 05:07:09.770912 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.752964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35413 (local address 127.8.215.3:58493)
0925 05:07:09.753753 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:09.753795 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:09.755489 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:09.756412 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:09.756424 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:09.757202 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:09.757833 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.757851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.759205 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.759210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.760644 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.760657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.761411 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.761416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.761606 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.762325 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:09.762351 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:09.764412 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:09.767111 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:09.767118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:09.767133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:09.767539 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:09.767844 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:09.767847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:09.767849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:09.768094 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:09.768545 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:09.768550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:09.768923 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 05:07:09.769186 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 05:07:09.769606 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":335,"threads_started":1}
I20260925 05:07:09.772324 11975 heartbeater.cc:347] Connected to a master server at 127.8.215.62:35413
I20260925 05:07:09.772624 11975 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:09.773363 11975 heartbeater.cc:511] Master 127.8.215.62:35413 requested a full tablet report, sending...
I20260925 05:07:09.774724 11519 ts_manager.cc:194] Registered new tserver with Master: 84bd6a28486d4fc08ae2a4d185b610d9 (127.8.215.3:43917)
I20260925 05:07:09.775470 11519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:58493
I20260925 05:07:09.784372 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:07:09.816542 11977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:09.814001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.8.215.62:35413)
0925 05:07:09.814163 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:09.814167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:09.814634 (+ 467us) server_negotiation.cc:407] Connection header received
0925 05:07:09.814834 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:09.814840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:09.814915 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:09.815022 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:09.816278 (+ 1256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40618: BlockingRecv error: recv got EOF from 127.0.0.1:40618 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:07:09.819535 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11580
W20260925 05:07:09.822363 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11580: Not authorized: Client connection negotiation failed: client connection to 127.8.215.1:36553: server requires authentication, but client does not have Kerberos credentials available
I20260925 05:07:09.822530 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 11580
I20260925 05:07:09.838464 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11714
W20260925 05:07:09.841176 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11714: Not authorized: Client connection negotiation failed: client connection to 127.8.215.2:39905: server requires authentication, but client does not have Kerberos credentials available
I20260925 05:07:09.841390 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 11714
I20260925 05:07:09.856949 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11847
W20260925 05:07:09.859788 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11847: Not authorized: Client connection negotiation failed: client connection to 127.8.215.3:43917: server requires authentication, but client does not have Kerberos credentials available
I20260925 05:07:09.859933 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 11847
I20260925 05:07:09.875488 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 11487
I20260925 05:07:09.876721 11549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:09.978358 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 11487
2026-09-25T05:07:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5865 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:10 dist-test-slave-wpdm krb5kdc[11993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:10 dist-test-slave-wpdm krb5kdc[11993](info): set up 2 sockets
Sep 25 05:07:10 dist-test-slave-wpdm krb5kdc[11993](info): commencing operation
krb5kdc: starting...
W20260925 05:07:11.069638 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:11 dist-test-slave-wpdm krb5kdc[11993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:11.272619 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:46573
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:11.597363 12006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:11.597713 12006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:11.597882 12006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:11.605165 12006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:11.605322 12006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:11.605427 12006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:11.605525 12006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:11.605608 12006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:11.615572 12006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:11.617138 12006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:11.618463 12006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:11.630270 12012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:11.630210 12014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:11.630245 12011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:11.631527 12006 server_base.cc:1034] running on GCE node
I20260925 05:07:11.632299 12006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:11.633852 12006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:11.635049 12006 hybrid_clock.cc:648] HybridClock initialized: now 1790312831635007 us; error 34 us; skew 500 ppm
I20260925 05:07:11.635413 12006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:11 dist-test-slave-wpdm krb5kdc[11993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312831, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:11.638602 12006 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:11.640064 12006 webserver.cc:460] Webserver started at http://127.8.215.62:41929/ using document root <none> and password file <none>
I20260925 05:07:11.640823 12006 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:11.640969 12006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:11.641283 12006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:11.644204 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "7c673d70d8a54c6e827038c7673e619c"
format_stamp: "Formatted at 2026-09-25 05:07:11 on dist-test-slave-wpdm"
server_key: "7f6d04e122d5c43165295b7c1475fa0f"
server_key_iv: "74f6ecbbb6f34dbe2c6bdaf590137e0c"
server_key_version: "encryptionkey@0"
I20260925 05:07:11.645066 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "7c673d70d8a54c6e827038c7673e619c"
format_stamp: "Formatted at 2026-09-25 05:07:11 on dist-test-slave-wpdm"
server_key: "7f6d04e122d5c43165295b7c1475fa0f"
server_key_iv: "74f6ecbbb6f34dbe2c6bdaf590137e0c"
server_key_version: "encryptionkey@0"
I20260925 05:07:11.650161 12006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:07:11.653667 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:11.655020 12006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 05:07:11.655206 12006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "7c673d70d8a54c6e827038c7673e619c"
format_stamp: "Formatted at 2026-09-25 05:07:11 on dist-test-slave-wpdm"
server_key: "7f6d04e122d5c43165295b7c1475fa0f"
server_key_iv: "74f6ecbbb6f34dbe2c6bdaf590137e0c"
server_key_version: "encryptionkey@0"
I20260925 05:07:11.655422 12006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:11.695087 12006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:11.699262 12006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:11.699579 12006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:11.717881 12006 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:46573
I20260925 05:07:11.717926 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:46573 every 8 connection(s)
I20260925 05:07:11.719298 12006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:11.722349 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12006
I20260925 05:07:11.722548 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:11.723390 9052 external_mini_cluster.cc:1428] Setting key 55472ecb08ffee1b4f0371563e5fd025
I20260925 05:07:11.723548 12073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:07:11 dist-test-slave-wpdm krb5kdc[11993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:11.738662 12073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c: Bootstrap starting.
I20260925 05:07:11.742167 12073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:11.743455 12073 log.cc:824] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:11.746738 12073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c: No bootstrap required, opened a new log
I20260925 05:07:11.746943 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:11.725273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.8.215.62:46573)
0925 05:07:11.726072 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:11.726084 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:11.726126 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:07:11.728095 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:11.728119 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:11.728509 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:11.728917 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:11.730602 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:11.731871 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:11.733638 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:11.734406 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:11.734543 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:11.739450 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:11.739458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:11.739474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:11.739515 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:11.742655 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:11.743485 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:11.743491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:11.743497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:11.743694 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:11.744127 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:11.744137 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:11.744142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:11.744920 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:11.745127 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:11.745588 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 05:07:11.745923 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":388,"threads_started":1}
I20260925 05:07:11.750591 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:07:11.757387 12073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c673d70d8a54c6e827038c7673e619c" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46573 } }
I20260925 05:07:11.757928 12073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:11.758052 12073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c673d70d8a54c6e827038c7673e619c, State: Initialized, Role: FOLLOWER
I20260925 05:07:11.758831 12073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c673d70d8a54c6e827038c7673e619c" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46573 } }
I20260925 05:07:11.759174 12073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:11.759351 12073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:11.759547 12073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:11.761991 12073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c673d70d8a54c6e827038c7673e619c" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46573 } }
I20260925 05:07:11.762554 12073 leader_election.cc:302] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c673d70d8a54c6e827038c7673e619c; no voters:
I20260925 05:07:11.763270 12073 leader_election.cc:288] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:11.763393 12079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:11.764992 12079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [term 1 LEADER]: Becoming Leader. State: Replica: 7c673d70d8a54c6e827038c7673e619c, State: Running, Role: LEADER
I20260925 05:07:11.765602 12079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c673d70d8a54c6e827038c7673e619c" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46573 } }
I20260925 05:07:11.765918 12073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:11.768846 12080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c673d70d8a54c6e827038c7673e619c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c673d70d8a54c6e827038c7673e619c" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46573 } } }
I20260925 05:07:11.769342 12080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:11.770594 12081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c673d70d8a54c6e827038c7673e619c. Latest consensus state: current_term: 1 leader_uuid: "7c673d70d8a54c6e827038c7673e619c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c673d70d8a54c6e827038c7673e619c" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46573 } } }
I20260925 05:07:11.770977 12081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:11.774598 12084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:11.781395 12084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:11.802239 12084 catalog_manager.cc:1347] Generated new cluster ID: 0149c7f56ab84f19a69b447428a4a787
I20260925 05:07:11.802389 12084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:11.814252 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:11.803986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.8.215.62:46573)
0925 05:07:11.804184 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:11.804190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:11.805453 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 05:07:11.805631 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:11.805637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:11.805715 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:11.805817 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:11.807315 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:11.808381 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:11.810333 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:11.810983 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:11.811105 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:11.812407 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:11.812413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:11.812417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:11.813325 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:11.813378 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:11.813417 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:11.813524 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:11.813735 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:07:11.813836 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:11.814610 12084 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:11.816304 12084 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:11.828377 12084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c673d70d8a54c6e827038c7673e619c: Generated new TSK 0
I20260925 05:07:11.829342 12084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:07:11.841887 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12006
I20260925 05:07:11.842949 12068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:11.933706 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12006
2026-09-25T05:07:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1950 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:12 dist-test-slave-wpdm krb5kdc[12107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:12 dist-test-slave-wpdm krb5kdc[12107](info): set up 2 sockets
Sep 25 05:07:12 dist-test-slave-wpdm krb5kdc[12107](info): commencing operation
krb5kdc: starting...
W20260925 05:07:13.025749 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:13 dist-test-slave-wpdm krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:13.223956 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:41345
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:41345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:13.551944 12120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:13.552285 12120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:13.552444 12120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:13.559723 12120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:13.559875 12120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:13.559983 12120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:13.560084 12120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:13.560154 12120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:13.570086 12120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:41345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:41345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:13.571465 12120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:13.572789 12120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:13.584432 12125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:13.584388 12126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:13.584460 12128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:13.585378 12120 server_base.cc:1034] running on GCE node
I20260925 05:07:13.586143 12120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:13.587546 12120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:13.588851 12120 hybrid_clock.cc:648] HybridClock initialized: now 1790312833588781 us; error 55 us; skew 500 ppm
I20260925 05:07:13.589202 12120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:13 dist-test-slave-wpdm krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312833, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:13.592880 12120 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:13.595211 12120 webserver.cc:460] Webserver started at http://127.8.215.62:39585/ using document root <none> and password file <none>
I20260925 05:07:13.596220 12120 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:13.596359 12120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:13.596786 12120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:13.600591 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "bbe5c338f8cb40aebad62ce7ebaf573e"
format_stamp: "Formatted at 2026-09-25 05:07:13 on dist-test-slave-wpdm"
server_key: "96f2f90dd18f797cfd2116f95b21cf1b"
server_key_iv: "7120d1913711d7883c06586d8296c4ee"
server_key_version: "encryptionkey@0"
I20260925 05:07:13.601548 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "bbe5c338f8cb40aebad62ce7ebaf573e"
format_stamp: "Formatted at 2026-09-25 05:07:13 on dist-test-slave-wpdm"
server_key: "96f2f90dd18f797cfd2116f95b21cf1b"
server_key_iv: "7120d1913711d7883c06586d8296c4ee"
server_key_version: "encryptionkey@0"
I20260925 05:07:13.607007 12120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:13.610620 12135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:13.612030 12120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:07:13.612211 12120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "bbe5c338f8cb40aebad62ce7ebaf573e"
format_stamp: "Formatted at 2026-09-25 05:07:13 on dist-test-slave-wpdm"
server_key: "96f2f90dd18f797cfd2116f95b21cf1b"
server_key_iv: "7120d1913711d7883c06586d8296c4ee"
server_key_version: "encryptionkey@0"
I20260925 05:07:13.612409 12120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:13.636993 12120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:13.641104 12120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:13.641378 12120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:13.660205 12120 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:41345
I20260925 05:07:13.660235 12186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:41345 every 8 connection(s)
I20260925 05:07:13.661686 12120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:13.662523 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12120
I20260925 05:07:13.662778 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:13.663508 9052 external_mini_cluster.cc:1428] Setting key bcd8d327fba55356d70b3cd3710be531
I20260925 05:07:13.666227 12187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:07:13 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:13.679764 12187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e: Bootstrap starting.
I20260925 05:07:13.683068 12187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:13.684222 12187 log.cc:824] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:13.686738 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:13.665487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.8.215.62:41345)
0925 05:07:13.666245 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:13.666257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:13.666297 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:07:13.667947 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:13.667974 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:13.668372 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:13.668771 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:13.670573 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:13.671996 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:13.673914 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:13.674713 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:13.674874 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:13.679581 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:13.679589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:13.679606 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:13.679647 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:13.682688 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:13.683441 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:13.683449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:13.683455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:13.683630 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:13.684000 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:13.684004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:13.684005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:13.684562 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:13.684975 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:13.685375 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 05:07:13.685782 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":354,"threads_started":1}
I20260925 05:07:13.687300 12187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e: No bootstrap required, opened a new log
I20260925 05:07:13.695070 12187 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41345 } }
I20260925 05:07:13.695575 12187 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:13.695681 12187 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe5c338f8cb40aebad62ce7ebaf573e, State: Initialized, Role: FOLLOWER
I20260925 05:07:13.696318 12187 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41345 } }
I20260925 05:07:13.696552 12187 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:13.696696 12187 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:13.696882 12187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:13.699071 12187 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41345 } }
I20260925 05:07:13.699605 12187 leader_election.cc:302] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe5c338f8cb40aebad62ce7ebaf573e; no voters:
I20260925 05:07:13.700166 12187 leader_election.cc:288] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:13.700294 12192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:13.700991 12192 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [term 1 LEADER]: Becoming Leader. State: Replica: bbe5c338f8cb40aebad62ce7ebaf573e, State: Running, Role: LEADER
I20260925 05:07:13.701543 12192 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41345 } }
I20260925 05:07:13.702209 12187 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:13.703999 12194 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe5c338f8cb40aebad62ce7ebaf573e. Latest consensus state: current_term: 1 leader_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41345 } } }
I20260925 05:07:13.704298 12194 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:13.705197 12197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:13.705072 12193 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe5c338f8cb40aebad62ce7ebaf573e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41345 } } }
I20260925 05:07:13.705408 12193 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:13.710665 12197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:13.719408 12197 catalog_manager.cc:1347] Generated new cluster ID: 2e7b23baee2d43cd8dc5908f2207128a
I20260925 05:07:13.719532 12197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:13.744307 12197 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:13.745801 12197 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:13.754733 12197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe5c338f8cb40aebad62ce7ebaf573e: Generated new TSK 0
I20260925 05:07:13.755621 12197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:07:13.810352 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41345
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:14.136873 12213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:14.137192 12213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:14.137353 12213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:14.144748 12213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:14.144898 12213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:14.145076 12213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:14.154836 12213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:14.156136 12213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:14.157507 12213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:14.171017 12221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:14.171044 12219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:14.171295 12218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:14.171854 12213 server_base.cc:1034] running on GCE node
I20260925 05:07:14.172375 12213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:14.173302 12213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:14.174494 12213 hybrid_clock.cc:648] HybridClock initialized: now 1790312834174450 us; error 36 us; skew 500 ppm
I20260925 05:07:14.174816 12213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:14 dist-test-slave-wpdm krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:14.184010 12213 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:07:14.185606 12213 webserver.cc:460] Webserver started at http://127.8.215.1:37537/ using document root <none> and password file <none>
I20260925 05:07:14.186326 12213 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:14.186466 12213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:14.186774 12213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:14.189859 12213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "6cb2ab1565d04f209a26ae5f0e42fac5"
format_stamp: "Formatted at 2026-09-25 05:07:14 on dist-test-slave-wpdm"
server_key: "ec3486b16f84f1e3342a8fc8f764c992"
server_key_iv: "8122cc60aa62d2b1be38ace1bbd00dfd"
server_key_version: "encryptionkey@0"
I20260925 05:07:14.190632 12213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "6cb2ab1565d04f209a26ae5f0e42fac5"
format_stamp: "Formatted at 2026-09-25 05:07:14 on dist-test-slave-wpdm"
server_key: "ec3486b16f84f1e3342a8fc8f764c992"
server_key_iv: "8122cc60aa62d2b1be38ace1bbd00dfd"
server_key_version: "encryptionkey@0"
I20260925 05:07:14.195694 12213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:07:14.199141 12228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:14.200487 12213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 05:07:14.200651 12213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "6cb2ab1565d04f209a26ae5f0e42fac5"
format_stamp: "Formatted at 2026-09-25 05:07:14 on dist-test-slave-wpdm"
server_key: "ec3486b16f84f1e3342a8fc8f764c992"
server_key_iv: "8122cc60aa62d2b1be38ace1bbd00dfd"
server_key_version: "encryptionkey@0"
I20260925 05:07:14.200929 12213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:14.215713 12213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:14.220122 12213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:14.220425 12213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:14.221402 12213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:14.222867 12213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:14.222965 12213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:14.223110 12213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:14.223178 12213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:14.256438 12213 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:36283
I20260925 05:07:14.256474 12340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:36283 every 8 connection(s)
I20260925 05:07:14.257966 12213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:14.261022 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12213
I20260925 05:07:14.261215 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:14.262065 9052 external_mini_cluster.cc:1428] Setting key c61eac9b45aedbc91e00a5e2dd4ee3b8
Sep 25 05:07:14 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:14.279125 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:14.260388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:48525 (local address 127.8.215.62:41345)
0925 05:07:14.260883 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:14.260890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:14.264416 (+ 3526us) server_negotiation.cc:407] Connection header received
0925 05:07:14.264478 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:14.264483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:14.264567 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:14.264684 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:14.266267 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.267418 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.269047 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.269693 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.269838 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:14.273309 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:14.273314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:14.273317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:14.273348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:14.275761 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:14.276463 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:14.276473 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:14.276477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:14.276586 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:14.277036 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:14.277042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:14.277045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:14.277426 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:14.277574 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:14.278529 (+ 955us) server_negotiation.cc:299] Negotiation successful
0925 05:07:14.278744 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":229,"threads_started":1}
I20260925 05:07:14.280119 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:14.261054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:41345 (local address 127.8.215.1:48525)
0925 05:07:14.262011 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:14.262072 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:14.264046 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:14.264979 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:14.264988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:14.265521 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:14.266090 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:14.266107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.267581 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.267584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:14.268910 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:14.268921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.269927 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.269935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:14.270183 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:14.270954 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:14.270975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:14.273091 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:14.275906 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:14.275914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:14.275932 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:14.276334 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:14.276693 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:14.276699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:14.276702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:14.276919 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:14.277574 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:14.277584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:14.278108 (+ 524us) client_negotiation.cc:769] Sending connection context
0925 05:07:14.278405 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 05:07:14.278950 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
I20260925 05:07:14.281749 12341 heartbeater.cc:347] Connected to a master server at 127.8.215.62:41345
I20260925 05:07:14.282080 12341 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:14.282835 12341 heartbeater.cc:511] Master 127.8.215.62:41345 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
I20260925 05:07:14.285187 12152 ts_manager.cc:194] Registered new tserver with Master: 6cb2ab1565d04f209a26ae5f0e42fac5 (127.8.215.1:36283)
I20260925 05:07:14.286612 12152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:48525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:07:14.308794 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41345
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:14.640260 12347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:14.640527 12347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:14.640619 12347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:14.647855 12347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:14.647969 12347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:14.648082 12347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:14.657894 12347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:14.659114 12347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:14.660388 12347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:14.673596 12353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:14.673730 12355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:14.674548 12352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:14.674670 12347 server_base.cc:1034] running on GCE node
I20260925 05:07:14.675416 12347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:14.676391 12347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:14.677645 12347 hybrid_clock.cc:648] HybridClock initialized: now 1790312834677589 us; error 46 us; skew 500 ppm
I20260925 05:07:14.678002 12347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:14 dist-test-slave-wpdm krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:14.684609 12347 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:07:14.686195 12347 webserver.cc:460] Webserver started at http://127.8.215.2:46177/ using document root <none> and password file <none>
I20260925 05:07:14.686975 12347 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:14.687139 12347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:14.687462 12347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:14.690513 12347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "e17a50b39de84ed09f66ada4527eac10"
format_stamp: "Formatted at 2026-09-25 05:07:14 on dist-test-slave-wpdm"
server_key: "0e5150e58a2bca3170e1f439a6ebe17c"
server_key_iv: "c6adfb19237f69261db57d932841d08f"
server_key_version: "encryptionkey@0"
I20260925 05:07:14.691270 12347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "e17a50b39de84ed09f66ada4527eac10"
format_stamp: "Formatted at 2026-09-25 05:07:14 on dist-test-slave-wpdm"
server_key: "0e5150e58a2bca3170e1f439a6ebe17c"
server_key_iv: "c6adfb19237f69261db57d932841d08f"
server_key_version: "encryptionkey@0"
I20260925 05:07:14.698830 12347 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260925 05:07:14.703611 12362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:14.705729 12347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260925 05:07:14.705927 12347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "e17a50b39de84ed09f66ada4527eac10"
format_stamp: "Formatted at 2026-09-25 05:07:14 on dist-test-slave-wpdm"
server_key: "0e5150e58a2bca3170e1f439a6ebe17c"
server_key_iv: "c6adfb19237f69261db57d932841d08f"
server_key_version: "encryptionkey@0"
I20260925 05:07:14.706336 12347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:14.743170 12347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:14.747291 12347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:14.747591 12347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:14.748587 12347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:14.749915 12347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:14.749986 12347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:14.750150 12347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:14.750201 12347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:14.782366 12347 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:34155
I20260925 05:07:14.782413 12474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:34155 every 8 connection(s)
I20260925 05:07:14.783644 12347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:14.788932 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12347
I20260925 05:07:14.789113 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:14.789566 9052 external_mini_cluster.cc:1428] Setting key 247b7acfa001e01b5acbde138cc1cb56
Sep 25 05:07:14 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:14.807247 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:14.786125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:35673 (local address 127.8.215.62:41345)
0925 05:07:14.786558 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:14.786565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:14.787608 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 05:07:14.789498 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:14.789504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:14.789602 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:14.789722 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:14.791537 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.792247 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.794913 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.795309 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.795418 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:14.800111 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:14.800120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:14.800125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:14.800174 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:14.803733 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:14.804483 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:14.804492 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:14.804498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:14.804670 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:14.805173 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:14.805181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:14.805187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:14.805578 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:14.805738 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:14.806528 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 05:07:14.806778 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260925 05:07:14.808287 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:14.786726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:41345 (local address 127.8.215.2:35673)
0925 05:07:14.787458 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:14.787495 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:14.789219 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:14.790032 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:14.790044 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:14.790604 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:14.791376 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:14.791397 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.792405 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.792413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:14.794777 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:14.794793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:14.795536 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:14.795546 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:14.795861 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:14.796713 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:14.796776 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:14.799860 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:14.803898 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:14.803906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:14.803919 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:14.804335 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:14.804806 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:14.804809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:14.804824 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:14.805047 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:14.805744 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:14.805753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:14.806111 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 05:07:14.806338 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 05:07:14.806893 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":276,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
I20260925 05:07:14.815236 12475 heartbeater.cc:347] Connected to a master server at 127.8.215.62:41345
I20260925 05:07:14.815552 12475 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:14.816407 12475 heartbeater.cc:511] Master 127.8.215.62:41345 requested a full tablet report, sending...
I20260925 05:07:14.817989 12152 ts_manager.cc:194] Registered new tserver with Master: e17a50b39de84ed09f66ada4527eac10 (127.8.215.2:34155)
I20260925 05:07:14.818799 12152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:35673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:07:14.839948 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41345
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:15.161541 12481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:15.161803 12481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:15.161892 12481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:15.169127 12481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:15.169225 12481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:15.169337 12481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:15.178767 12481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:15.179971 12481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:15.181288 12481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:15.194226 12487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:15.194375 12486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:15.194339 12489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:15.195102 12481 server_base.cc:1034] running on GCE node
I20260925 05:07:15.195609 12481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:15.196584 12481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:15.197772 12481 hybrid_clock.cc:648] HybridClock initialized: now 1790312835197745 us; error 55 us; skew 500 ppm
I20260925 05:07:15.198050 12481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:15.201102 12481 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:07:15.202512 12481 webserver.cc:460] Webserver started at http://127.8.215.3:41117/ using document root <none> and password file <none>
I20260925 05:07:15.203243 12481 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:15.203369 12481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:15.203621 12481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:15.206651 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "400f3cece8c74ef1baa868e642dbd4d7"
format_stamp: "Formatted at 2026-09-25 05:07:15 on dist-test-slave-wpdm"
server_key: "73b051cd90d65cd4a70068bd4629e074"
server_key_iv: "835e80c84cd5d1ad3f179795fc90dcc1"
server_key_version: "encryptionkey@0"
I20260925 05:07:15.207490 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "400f3cece8c74ef1baa868e642dbd4d7"
format_stamp: "Formatted at 2026-09-25 05:07:15 on dist-test-slave-wpdm"
server_key: "73b051cd90d65cd4a70068bd4629e074"
server_key_iv: "835e80c84cd5d1ad3f179795fc90dcc1"
server_key_version: "encryptionkey@0"
I20260925 05:07:15.212708 12481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:07:15.216143 12496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:15.217592 12481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:07:15.217721 12481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "400f3cece8c74ef1baa868e642dbd4d7"
format_stamp: "Formatted at 2026-09-25 05:07:15 on dist-test-slave-wpdm"
server_key: "73b051cd90d65cd4a70068bd4629e074"
server_key_iv: "835e80c84cd5d1ad3f179795fc90dcc1"
server_key_version: "encryptionkey@0"
I20260925 05:07:15.217882 12481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:15.239596 12481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:15.243589 12481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:15.243837 12481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:15.244781 12481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:15.246057 12481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:15.246129 12481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:15.246208 12481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:15.246251 12481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:15.279081 12481 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:34391
I20260925 05:07:15.279129 12608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:34391 every 8 connection(s)
I20260925 05:07:15.280406 12481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:15.287815 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12481
I20260925 05:07:15.288043 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:15.288476 9052 external_mini_cluster.cc:1428] Setting key 599a7be7bafc76fe8d2a42976c03ca5e
I20260925 05:07:15.289511 12341 heartbeater.cc:503] Master 127.8.215.62:41345 was elected leader, sending a full tablet report...
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:15.300073 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:15.283118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:43571 (local address 127.8.215.62:41345)
0925 05:07:15.283296 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:15.283301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:15.284462 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 05:07:15.286353 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:15.286357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:15.286424 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:15.286515 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:15.288064 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.288993 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.291186 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.291740 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.291878 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:15.295146 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:15.295151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:15.295154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:15.295183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:15.297558 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.298215 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.298219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.298221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.298325 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.298680 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.298683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.298685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.298950 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:15.299052 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:15.299684 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 05:07:15.299844 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:07:15.301246 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:15.283585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:41345 (local address 127.8.215.3:43571)
0925 05:07:15.284287 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:15.284335 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:15.286090 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:15.286724 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:15.286732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:15.287243 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:15.287805 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:15.287831 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.289140 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.289148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:15.291031 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:15.291045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.291951 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.291956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:15.292155 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:15.292811 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:15.292852 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:15.294967 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:15.297680 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:15.297686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:15.297701 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:15.298118 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:15.298418 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:15.298421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:15.298423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:15.298592 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:15.299051 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:15.299057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:15.299374 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 05:07:15.299599 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 05:07:15.299998 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":273,"threads_started":1}
I20260925 05:07:15.302878 12609 heartbeater.cc:347] Connected to a master server at 127.8.215.62:41345
I20260925 05:07:15.303211 12609 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:15.304137 12609 heartbeater.cc:511] Master 127.8.215.62:41345 requested a full tablet report, sending...
I20260925 05:07:15.305634 12151 ts_manager.cc:194] Registered new tserver with Master: 400f3cece8c74ef1baa868e642dbd4d7 (127.8.215.3:34391)
I20260925 05:07:15.306501 12151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:43571
I20260925 05:07:15.316345 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:15.363900 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:15.350702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.8.215.62:41345)
0925 05:07:15.350872 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:15.350876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:15.351341 (+ 465us) server_negotiation.cc:407] Connection header received
0925 05:07:15.351472 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:15.351476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:15.351553 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:15.351644 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:15.352885 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.353579 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.354880 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.355274 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.355373 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:15.359160 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:15.359164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:15.359167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:15.359199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:15.361734 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.362287 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.362290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.362292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.362450 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.362746 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.362749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.362751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.363036 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:15.363227 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:15.363424 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:07:15.363573 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 05:07:15.364493 12151 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41922
I20260925 05:07:15.366775 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12213
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:15.381626 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:15.367697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.8.215.1:36283)
0925 05:07:15.368300 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:15.368307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:15.368331 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:07:15.368382 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:15.368388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:15.368591 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:15.368709 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:15.369824 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.370890 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.372009 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.372541 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.372634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:15.376304 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:15.376315 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:15.376329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:15.376358 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:15.379168 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.379737 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.379744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.379746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.379853 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.380270 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.380273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.380274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.380582 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:15.380853 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:15.381179 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 05:07:15.381361 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":319,"threads_started":1}
W20260925 05:07:15.382974 12336 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54488
W20260925 05:07:15.384115 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12213: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:07:15.384415 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12213
I20260925 05:07:15.402208 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12347
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:07:15.418394 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:15.403065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.8.215.2:34155)
0925 05:07:15.403670 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:15.403677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:15.403703 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:07:15.403764 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:15.403770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:15.404017 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:15.404142 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:15.405523 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.406496 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.407848 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.408454 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.408551 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:15.412764 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:15.412777 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:15.412796 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:15.412839 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:15.415663 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.416210 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.416218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.416220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.416394 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.416689 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.416692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.416694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.417124 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:15.417442 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:15.417787 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 05:07:15.418056 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":321,"threads_started":1}
W20260925 05:07:15.419598 12470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50014
W20260925 05:07:15.420680 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:07:15.420868 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12347
I20260925 05:07:15.436424 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12481
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:07:15.452028 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:15.437430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.8.215.3:34391)
0925 05:07:15.438037 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:15.438044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:15.438071 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:07:15.438128 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:15.438135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:15.438339 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:15.438519 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:15.439841 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.440990 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.442210 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:15.442772 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:15.442911 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:15.447132 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:15.447144 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:15.447158 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:15.447196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:15.449802 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.450366 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.450373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.450375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.450496 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:15.450779 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:15.450783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:15.450784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:15.451090 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:15.451301 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:15.451562 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 05:07:15.451748 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":314,"threads_started":1}
W20260925 05:07:15.453367 12604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33590
W20260925 05:07:15.454437 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:07:15.454596 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12481
I20260925 05:07:15.469813 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12120
I20260925 05:07:15.470927 12182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:15.571074 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12120
2026-09-25T05:07:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3642 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12626](info): set up 2 sockets
Sep 25 05:07:15 dist-test-slave-wpdm krb5kdc[12626](info): commencing operation
krb5kdc: starting...
W20260925 05:07:16.685575 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:16 dist-test-slave-wpdm krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:16.884655 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43923
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:17.200197 12639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:17.200541 12639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:17.200700 12639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:17.207859 12639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:17.208007 12639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:17.208118 12639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:17.208215 12639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:17.208299 12639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:17.217975 12639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:17.219262 12639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:17.220523 12639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:17.232134 12645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:17.232187 12644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:17.232188 12647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:17.233110 12639 server_base.cc:1034] running on GCE node
I20260925 05:07:17.233860 12639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:17.235284 12639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:17.236508 12639 hybrid_clock.cc:648] HybridClock initialized: now 1790312837236460 us; error 44 us; skew 500 ppm
I20260925 05:07:17.236899 12639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:17 dist-test-slave-wpdm krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:17.239816 12639 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:17.241262 12639 webserver.cc:460] Webserver started at http://127.8.215.62:45517/ using document root <none> and password file <none>
I20260925 05:07:17.241892 12639 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:17.242041 12639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:17.242343 12639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:17.245225 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "3bc2f31495d647e68244679af7496a36"
format_stamp: "Formatted at 2026-09-25 05:07:17 on dist-test-slave-wpdm"
server_key: "c7154f82324ef8df0cba754f701c0178"
server_key_iv: "a5e5a5f648feb4c833699ee588c9c044"
server_key_version: "encryptionkey@0"
I20260925 05:07:17.245971 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "3bc2f31495d647e68244679af7496a36"
format_stamp: "Formatted at 2026-09-25 05:07:17 on dist-test-slave-wpdm"
server_key: "c7154f82324ef8df0cba754f701c0178"
server_key_iv: "a5e5a5f648feb4c833699ee588c9c044"
server_key_version: "encryptionkey@0"
I20260925 05:07:17.250878 12639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:17.254120 12654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:17.255347 12639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:07:17.255523 12639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "3bc2f31495d647e68244679af7496a36"
format_stamp: "Formatted at 2026-09-25 05:07:17 on dist-test-slave-wpdm"
server_key: "c7154f82324ef8df0cba754f701c0178"
server_key_iv: "a5e5a5f648feb4c833699ee588c9c044"
server_key_version: "encryptionkey@0"
I20260925 05:07:17.255708 12639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:17.266000 12639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:17.269909 12639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:17.270196 12639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:17.287882 12639 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43923
I20260925 05:07:17.287910 12705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43923 every 8 connection(s)
I20260925 05:07:17.289336 12639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:17.292559 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12639
I20260925 05:07:17.292800 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:17.293351 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:17.293588 9052 external_mini_cluster.cc:1428] Setting key ed3f65a81864d2f526905f655a362b52
I20260925 05:07:17.305467 12706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36: Bootstrap starting.
Sep 25 05:07:17 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:17.308708 12706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:17.309927 12706 log.cc:824] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:17.312682 12706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36: No bootstrap required, opened a new log
I20260925 05:07:17.315187 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:17.295596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.8.215.62:43923)
0925 05:07:17.296337 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:17.296351 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:17.296391 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:07:17.298051 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:17.298073 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:17.298464 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:17.298862 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:17.300308 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:17.301600 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:17.302837 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:17.303534 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:17.303732 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:17.308038 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:17.308045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:17.308059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:17.308096 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:17.311206 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:17.311913 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:17.311920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:17.311927 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:17.312078 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:17.312447 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:17.312452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:17.312460 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:17.313056 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:17.313379 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:17.313770 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 05:07:17.314179 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":331,"threads_started":1}
I20260925 05:07:17.320394 12706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc2f31495d647e68244679af7496a36" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43923 } }
I20260925 05:07:17.320722 12706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:17.320879 12706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc2f31495d647e68244679af7496a36, State: Initialized, Role: FOLLOWER
I20260925 05:07:17.321484 12706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc2f31495d647e68244679af7496a36" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43923 } }
I20260925 05:07:17.321794 12706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:17.321936 12706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:17.322121 12706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:17.324204 12706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc2f31495d647e68244679af7496a36" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43923 } }
I20260925 05:07:17.324687 12706 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc2f31495d647e68244679af7496a36; no voters:
I20260925 05:07:17.325385 12706 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:17.325534 12711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:17.326246 12711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc2f31495d647e68244679af7496a36, State: Running, Role: LEADER
I20260925 05:07:17.326730 12711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc2f31495d647e68244679af7496a36" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43923 } }
I20260925 05:07:17.327375 12706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:17.329936 12713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc2f31495d647e68244679af7496a36. Latest consensus state: current_term: 1 leader_uuid: "3bc2f31495d647e68244679af7496a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc2f31495d647e68244679af7496a36" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43923 } } }
I20260925 05:07:17.330384 12713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:17.331496 12712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bc2f31495d647e68244679af7496a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc2f31495d647e68244679af7496a36" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43923 } } }
I20260925 05:07:17.331823 12712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:17.333191 12721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:17.337962 12721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:17.347594 12721 catalog_manager.cc:1347] Generated new cluster ID: e127f8dcb9d7494b94378a76016a1949
I20260925 05:07:17.347738 12721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:17.356828 12721 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:17.358184 12721 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:17.367858 12721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc2f31495d647e68244679af7496a36: Generated new TSK 0
I20260925 05:07:17.368676 12721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:07:17.419849 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43923
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:17.748042 12732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:17.748339 12732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:17.748437 12732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:17.756868 12732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:17.757001 12732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:17.757139 12732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:17.768376 12732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:17.769759 12732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:17.771102 12732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:17.785390 12740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:17.785426 12738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:17.785697 12737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:17.786531 12732 server_base.cc:1034] running on GCE node
I20260925 05:07:17.787149 12732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:17.788177 12732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:17.789382 12732 hybrid_clock.cc:648] HybridClock initialized: now 1790312837789338 us; error 45 us; skew 500 ppm
I20260925 05:07:17.789705 12732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:17 dist-test-slave-wpdm krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:17.792923 12732 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:07:17.794445 12732 webserver.cc:460] Webserver started at http://127.8.215.1:34593/ using document root <none> and password file <none>
I20260925 05:07:17.795120 12732 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:17.795261 12732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:17.795567 12732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:17.798550 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "ee6e84573a12498a93027e3831e9e2ae"
format_stamp: "Formatted at 2026-09-25 05:07:17 on dist-test-slave-wpdm"
server_key: "e9712537023852d39f8d258c0e508260"
server_key_iv: "3c940f8f3a08fdbcc0c704b18d4e8bce"
server_key_version: "encryptionkey@0"
I20260925 05:07:17.799325 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "ee6e84573a12498a93027e3831e9e2ae"
format_stamp: "Formatted at 2026-09-25 05:07:17 on dist-test-slave-wpdm"
server_key: "e9712537023852d39f8d258c0e508260"
server_key_iv: "3c940f8f3a08fdbcc0c704b18d4e8bce"
server_key_version: "encryptionkey@0"
I20260925 05:07:17.804467 12732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 05:07:17.808044 12747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:17.809509 12732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:17.809691 12732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "ee6e84573a12498a93027e3831e9e2ae"
format_stamp: "Formatted at 2026-09-25 05:07:17 on dist-test-slave-wpdm"
server_key: "e9712537023852d39f8d258c0e508260"
server_key_iv: "3c940f8f3a08fdbcc0c704b18d4e8bce"
server_key_version: "encryptionkey@0"
I20260925 05:07:17.809919 12732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:17.826162 12732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:17.830608 12732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:17.830899 12732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:17.831820 12732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:17.833158 12732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:17.833256 12732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:17.833402 12732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:17.833477 12732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:17.866160 12732 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:34193
I20260925 05:07:17.866204 12859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:34193 every 8 connection(s)
I20260925 05:07:17.867456 12732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:17.869882 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12732
I20260925 05:07:17.870105 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:17.870873 9052 external_mini_cluster.cc:1428] Setting key c35b0f1d281278f9b5a70fa6247aa84a
Sep 25 05:07:17 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:17.888324 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:17.870085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:47397 (local address 127.8.215.62:43923)
0925 05:07:17.870474 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:17.870479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:17.871761 (+ 1282us) server_negotiation.cc:407] Connection header received
0925 05:07:17.873943 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:17.873948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:17.874055 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:17.874168 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:17.875622 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:17.876329 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:17.877964 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:17.878388 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:17.878494 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:17.882523 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:17.882528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:17.882531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:17.882561 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:17.885037 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:17.885844 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:17.885852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:17.885857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:17.886003 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:17.886423 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:17.886431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:17.886436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:17.886846 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:17.887016 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:17.887676 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 05:07:17.887916 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":209,"threads_started":1}
I20260925 05:07:17.889148 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:17.870784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43923 (local address 127.8.215.1:47397)
0925 05:07:17.871618 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:17.871675 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:17.873684 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:17.874402 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:17.874410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:17.874914 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:17.875461 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:17.875478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:17.876503 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:17.876507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:17.877809 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:17.877828 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:17.878617 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:17.878626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:17.878949 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:17.879675 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:17.879708 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:17.882273 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:17.885179 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:17.885188 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:17.885204 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:17.885686 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:17.886120 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:17.886123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:17.886125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:17.886297 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:17.886968 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:17.886973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:17.887268 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 05:07:17.887482 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 05:07:17.887891 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":329,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
I20260925 05:07:17.890743 12860 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43923
I20260925 05:07:17.891067 12860 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:17.891755 12860 heartbeater.cc:511] Master 127.8.215.62:43923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
I20260925 05:07:17.893837 12671 ts_manager.cc:194] Registered new tserver with Master: ee6e84573a12498a93027e3831e9e2ae (127.8.215.1:34193)
I20260925 05:07:17.895248 12671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:47397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:07:17.916509 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43923
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:18.231052 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:18.231334 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:18.231509 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:18.238787 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:18.238909 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:18.239084 12866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:18.248560 12866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:18.249791 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:18.251017 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:18.263803 12872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:18.263851 12871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:18.263827 12874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:18.265002 12866 server_base.cc:1034] running on GCE node
I20260925 05:07:18.265564 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:18.266515 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:18.267750 12866 hybrid_clock.cc:648] HybridClock initialized: now 1790312838267700 us; error 48 us; skew 500 ppm
I20260925 05:07:18.268055 12866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:18.271092 12866 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:07:18.272392 12866 webserver.cc:460] Webserver started at http://127.8.215.2:45233/ using document root <none> and password file <none>
I20260925 05:07:18.273222 12866 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:18.273365 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:18.273661 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:18.276428 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "d1ef6e1a235647f2b33d09eac402db74"
format_stamp: "Formatted at 2026-09-25 05:07:18 on dist-test-slave-wpdm"
server_key: "bf7fa804b1e35060c4a31714dd55fd07"
server_key_iv: "7d443cb50b3510e319dea220ece294f5"
server_key_version: "encryptionkey@0"
I20260925 05:07:18.277237 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "d1ef6e1a235647f2b33d09eac402db74"
format_stamp: "Formatted at 2026-09-25 05:07:18 on dist-test-slave-wpdm"
server_key: "bf7fa804b1e35060c4a31714dd55fd07"
server_key_iv: "7d443cb50b3510e319dea220ece294f5"
server_key_version: "encryptionkey@0"
I20260925 05:07:18.282045 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 05:07:18.285418 12881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:18.286715 12866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:18.286880 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "d1ef6e1a235647f2b33d09eac402db74"
format_stamp: "Formatted at 2026-09-25 05:07:18 on dist-test-slave-wpdm"
server_key: "bf7fa804b1e35060c4a31714dd55fd07"
server_key_iv: "7d443cb50b3510e319dea220ece294f5"
server_key_version: "encryptionkey@0"
I20260925 05:07:18.287091 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:18.300614 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:18.304903 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:18.305240 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:18.306165 12866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:18.307554 12866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:18.307673 12866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:18.307816 12866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:18.307885 12866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:18.340569 12866 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:34485
I20260925 05:07:18.340603 12993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:34485 every 8 connection(s)
I20260925 05:07:18.341876 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:18.344673 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12866
I20260925 05:07:18.344893 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:18.345290 9052 external_mini_cluster.cc:1428] Setting key 9555822e9bc97a4aee893d3ef77fd72d
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:18.363282 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.344484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:57075 (local address 127.8.215.62:43923)
0925 05:07:18.344688 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:18.344694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:18.346005 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 05:07:18.347862 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:18.347869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:18.347954 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:18.348062 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:18.349535 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.350662 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.352133 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.352797 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.352901 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.356674 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:18.356683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:18.356691 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:18.356751 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:18.360391 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.361137 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.361149 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.361152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.361273 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.361688 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.361693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.361696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.361951 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:18.362111 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:18.362731 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 05:07:18.362942 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:07:18.364230 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.345054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43923 (local address 127.8.215.2:57075)
0925 05:07:18.345859 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:18.345903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:18.347600 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:18.348251 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:18.348259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:18.348825 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:18.349374 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:18.349390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.350803 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.350806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:18.351994 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:18.352005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.353030 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.353038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:18.353357 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.354151 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:18.354173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:18.356447 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:18.360529 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:18.360536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:18.360549 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:18.361015 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:18.361382 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:18.361388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:18.361392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:18.361567 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:18.362075 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:18.362084 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:18.362395 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 05:07:18.362644 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 05:07:18.363166 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
I20260925 05:07:18.365660 12994 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43923
I20260925 05:07:18.365971 12994 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:18.366652 12994 heartbeater.cc:511] Master 127.8.215.62:43923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
I20260925 05:07:18.368165 12671 ts_manager.cc:194] Registered new tserver with Master: d1ef6e1a235647f2b33d09eac402db74 (127.8.215.2:34485)
I20260925 05:07:18.369079 12671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:57075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:07:18.390287 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43923
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:18.706219 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:18.706526 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:18.706687 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:18.713900 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:18.714061 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:18.714226 12999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:18.723698 12999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:18.724972 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:18.726253 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:18.739382 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:18.739486 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:18.739514 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:18.740324 12999 server_base.cc:1034] running on GCE node
I20260925 05:07:18.740983 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:18.741887 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:18.743088 12999 hybrid_clock.cc:648] HybridClock initialized: now 1790312838743042 us; error 45 us; skew 500 ppm
I20260925 05:07:18.743392 12999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:18.746417 12999 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:07:18.747823 12999 webserver.cc:460] Webserver started at http://127.8.215.3:45011/ using document root <none> and password file <none>
I20260925 05:07:18.748443 12999 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:18.748539 12999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:18.748854 12999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:18.751577 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "36f0f416e0ff4f0ea6d852d09fdb1ead"
format_stamp: "Formatted at 2026-09-25 05:07:18 on dist-test-slave-wpdm"
server_key: "2910a5a906c069671b3165415488950a"
server_key_iv: "4326d7f6ec9e94b1fdc5add0d3778a97"
server_key_version: "encryptionkey@0"
I20260925 05:07:18.752260 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "36f0f416e0ff4f0ea6d852d09fdb1ead"
format_stamp: "Formatted at 2026-09-25 05:07:18 on dist-test-slave-wpdm"
server_key: "2910a5a906c069671b3165415488950a"
server_key_iv: "4326d7f6ec9e94b1fdc5add0d3778a97"
server_key_version: "encryptionkey@0"
I20260925 05:07:18.757150 12999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 05:07:18.760366 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:18.761653 12999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:18.761801 12999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "36f0f416e0ff4f0ea6d852d09fdb1ead"
format_stamp: "Formatted at 2026-09-25 05:07:18 on dist-test-slave-wpdm"
server_key: "2910a5a906c069671b3165415488950a"
server_key_iv: "4326d7f6ec9e94b1fdc5add0d3778a97"
server_key_version: "encryptionkey@0"
I20260925 05:07:18.761987 12999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:18.783543 12999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:18.788211 12999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:18.788533 12999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:18.789541 12999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:18.790886 12999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:18.791002 12999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:18.791149 12999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:18.791215 12999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:18.821929 12999 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:35523
I20260925 05:07:18.821983 13126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:35523 every 8 connection(s)
I20260925 05:07:18.823230 12999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:18.828477 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 12999
I20260925 05:07:18.828668 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:18.829193 9052 external_mini_cluster.cc:1428] Setting key 033a8f832cea434d311b4f6b7ea2bf20
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:18.843899 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.825756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:57607 (local address 127.8.215.62:43923)
0925 05:07:18.825929 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:18.825933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:18.827256 (+ 1323us) server_negotiation.cc:407] Connection header received
0925 05:07:18.829988 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:18.829996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:18.830083 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:18.830187 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:18.831792 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.832628 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.834202 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.834600 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.834692 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.838210 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:18.838215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:18.838218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:18.838247 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:18.840690 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.841379 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.841383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.841385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.841487 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.842002 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.842006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.842008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.842337 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:18.842455 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:18.843352 (+ 897us) server_negotiation.cc:299] Negotiation successful
0925 05:07:18.843567 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:18.845021 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.826240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43923 (local address 127.8.215.3:57607)
0925 05:07:18.827097 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:18.827145 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:18.829661 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:18.830389 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:18.830396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:18.831094 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:18.831638 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:18.831655 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.832801 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.832805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:18.834082 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:18.834092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.834795 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.834799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:18.835016 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.835609 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:18.835633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:18.838024 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:18.840849 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:18.840856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:18.840869 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:18.841265 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:18.841622 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:18.841626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:18.841629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:18.841851 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:18.842482 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:18.842489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:18.842885 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 05:07:18.843194 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 05:07:18.843677 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":311,"threads_started":1}
I20260925 05:07:18.846243 13127 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43923
I20260925 05:07:18.846555 13127 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:18.847242 13127 heartbeater.cc:511] Master 127.8.215.62:43923 requested a full tablet report, sending...
I20260925 05:07:18.848510 12671 ts_manager.cc:194] Registered new tserver with Master: 36f0f416e0ff4f0ea6d852d09fdb1ead (127.8.215.3:35523)
I20260925 05:07:18.849270 12671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:57607
I20260925 05:07:18.856534 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 05:07:18.897969 12860 heartbeater.cc:503] Master 127.8.215.62:43923 was elected leader, sending a full tablet report...
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:18.905622 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.891072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.8.215.1:34193)
0925 05:07:18.891764 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:18.891771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:18.891796 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:07:18.891859 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:18.891866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:18.892072 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:18.892238 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:18.893637 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.894643 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.895887 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.896452 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.896584 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.900329 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:18.900340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:18.900356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:18.900390 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:18.902957 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.903590 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.903597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.903599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.903711 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.904076 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.904082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.904085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.904401 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:18.904689 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:18.905018 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 05:07:18.905279 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":288,"threads_started":1}
Sep 25 05:07:18 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:18.922057 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.909644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.8.215.62:43923)
0925 05:07:18.909860 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:18.909864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:18.909937 (+ 73us) server_negotiation.cc:407] Connection header received
0925 05:07:18.910058 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:18.910062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:18.910120 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:18.910197 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:18.911312 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.911999 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.913256 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.913673 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.913784 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.917571 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:18.917575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:18.917578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:18.917608 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:18.919812 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.920373 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.920377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.920379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.920501 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.920830 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.920833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.920835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.921129 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:18.921297 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:18.921543 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 05:07:18.921702 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 05:07:18.922629 12671 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178
I20260925 05:07:18.923590 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12732
I20260925 05:07:18.935894 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:18.924394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.8.215.1:34193)
0925 05:07:18.924540 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:18.924544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:18.924862 (+ 318us) server_negotiation.cc:407] Connection header received
0925 05:07:18.924973 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:18.924976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:18.925035 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:18.925110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:18.926174 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.926815 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.927957 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:18.928454 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:18.928550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:18.930614 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:18.930618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:18.930621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:18.930648 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:18.933186 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.933799 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.933809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.933811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.933969 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:18.934344 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:18.934349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:18.934352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:18.934585 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:18.934833 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:18.935221 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 05:07:18.935472 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:07:18.937227 12855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:19.030823 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12732
I20260925 05:07:19.047529 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12866
Sep 25 05:07:19 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:07:19.063380 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:19.048430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.8.215.2:34485)
0925 05:07:19.049072 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:19.049079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:19.049103 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:07:19.049160 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:19.049166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:19.049367 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:19.049487 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:19.050764 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:19.051726 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:19.053236 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:19.053812 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:19.053918 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:19.057902 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:19.057914 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:19.057929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:19.057959 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:19.061188 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:19.061733 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:19.061739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:19.061742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:19.061846 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:19.062152 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:19.062155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:19.062157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:19.062471 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:19.062691 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:19.062919 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:07:19.063103 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":356,"threads_started":1}
I20260925 05:07:19.064710 12989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:19.145236 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12866
I20260925 05:07:19.161439 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12999
Sep 25 05:07:19 dist-test-slave-wpdm krb5kdc[12626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:07:19.177568 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:19.162416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.8.215.3:35523)
0925 05:07:19.163025 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:19.163032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:19.163060 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:07:19.163116 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:19.163122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:19.163321 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:19.163448 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:19.164630 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:19.165709 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:19.167313 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:19.167995 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:19.168133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:19.172279 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:19.172290 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:19.172304 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:19.172339 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:19.175194 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:19.175752 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:19.175758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:19.175760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:19.175891 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:19.176190 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:19.176193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:19.176195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:19.176497 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:19.176772 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:19.176988 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 05:07:19.177174 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":320,"threads_started":1}
I20260925 05:07:19.178822 13122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:19.262089 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12999
I20260925 05:07:19.278115 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 12639
I20260925 05:07:19.279460 12701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:19.378991 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 12639
2026-09-25T05:07:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3808 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:19 dist-test-slave-wpdm krb5kdc[13146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:19 dist-test-slave-wpdm krb5kdc[13146](info): set up 2 sockets
Sep 25 05:07:19 dist-test-slave-wpdm krb5kdc[13146](info): commencing operation
krb5kdc: starting...
W20260925 05:07:20.481669 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:20 dist-test-slave-wpdm krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:20.674422 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:45575
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:21.004383 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:21.004724 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:21.004928 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:21.012106 13159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:21.012326 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:21.012455 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:21.012557 13159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:21.012658 13159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:21.022485 13159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:45575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:21.023910 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:21.025305 13159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:21.037189 13165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:21.037194 13164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:21.037173 13167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:21.038084 13159 server_base.cc:1034] running on GCE node
I20260925 05:07:21.038889 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:21.040211 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:21.041405 13159 hybrid_clock.cc:648] HybridClock initialized: now 1790312841041373 us; error 26 us; skew 500 ppm
I20260925 05:07:21.041759 13159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:21 dist-test-slave-wpdm krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312841, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:21.045006 13159 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:21.046083 13159 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:21.046216 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:21.046535 13159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:21.049425 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "72ac9f1dac7f4bfa9857b7d3306191d8"
format_stamp: "Formatted at 2026-09-25 05:07:21 on dist-test-slave-wpdm"
server_key: "ef5b9fbe199735412db983eec4c10bd0"
server_key_iv: "86717f9fcb30ea1775d541d762b49eaa"
server_key_version: "encryptionkey@0"
I20260925 05:07:21.050192 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "72ac9f1dac7f4bfa9857b7d3306191d8"
format_stamp: "Formatted at 2026-09-25 05:07:21 on dist-test-slave-wpdm"
server_key: "ef5b9fbe199735412db983eec4c10bd0"
server_key_iv: "86717f9fcb30ea1775d541d762b49eaa"
server_key_version: "encryptionkey@0"
I20260925 05:07:21.055054 13159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:21.058434 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:21.059689 13159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:07:21.059861 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "72ac9f1dac7f4bfa9857b7d3306191d8"
format_stamp: "Formatted at 2026-09-25 05:07:21 on dist-test-slave-wpdm"
server_key: "ef5b9fbe199735412db983eec4c10bd0"
server_key_iv: "86717f9fcb30ea1775d541d762b49eaa"
server_key_version: "encryptionkey@0"
I20260925 05:07:21.060081 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:21.076568 13159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:21.080230 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:21.080492 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:21.098238 13159 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:45575
I20260925 05:07:21.098258 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:45575 every 8 connection(s)
I20260925 05:07:21.099133 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:21.102591 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13159
I20260925 05:07:21.102823 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:21.102900 13225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:21.103601 9052 external_mini_cluster.cc:1428] Setting key c571b59433bd1f6b0793a9c4eeeb21fa
I20260925 05:07:21.117383 13225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8: Bootstrap starting.
Sep 25 05:07:21 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:21.121129 13225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:21.122426 13225 log.cc:824] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:21.125636 13225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8: No bootstrap required, opened a new log
I20260925 05:07:21.127132 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:21.105664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.8.215.62:45575)
0925 05:07:21.106480 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:21.106492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:21.106531 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:07:21.108027 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:21.108049 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:21.108409 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:21.108781 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:21.110534 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:21.111770 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:21.113540 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:21.114238 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:21.114365 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:21.118911 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:21.118924 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:21.118945 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:21.118994 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:21.122755 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:21.123562 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:21.123567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:21.123573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:21.123773 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:21.124153 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:21.124157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:21.124160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:21.124841 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:21.125138 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:21.125456 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 05:07:21.125846 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":404,"threads_started":1}
I20260925 05:07:21.133953 13225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45575 } }
I20260925 05:07:21.134276 13225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:21.134346 13225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ac9f1dac7f4bfa9857b7d3306191d8, State: Initialized, Role: FOLLOWER
I20260925 05:07:21.134791 13225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45575 } }
I20260925 05:07:21.134976 13225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:21.135068 13225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:21.135175 13225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:21.137476 13225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45575 } }
I20260925 05:07:21.137931 13225 leader_election.cc:302] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72ac9f1dac7f4bfa9857b7d3306191d8; no voters:
I20260925 05:07:21.138447 13225 leader_election.cc:288] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:21.138559 13230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:21.139384 13230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [term 1 LEADER]: Becoming Leader. State: Replica: 72ac9f1dac7f4bfa9857b7d3306191d8, State: Running, Role: LEADER
I20260925 05:07:21.139899 13230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45575 } }
I20260925 05:07:21.140625 13225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:21.143114 13231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45575 } } }
I20260925 05:07:21.143582 13231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:21.143111 13232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72ac9f1dac7f4bfa9857b7d3306191d8. Latest consensus state: current_term: 1 leader_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac9f1dac7f4bfa9857b7d3306191d8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45575 } } }
I20260925 05:07:21.144104 13232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:21.145372 13237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:21.150769 13237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:21.159164 13237 catalog_manager.cc:1347] Generated new cluster ID: f092480220b641eba2a5c262a164783d
I20260925 05:07:21.159304 13237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:21.178303 13237 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:21.180194 13237 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:21.191128 13237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8: Generated new TSK 0
I20260925 05:07:21.192155 13237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:07:21.250530 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45575
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:21.572242 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:21.572553 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:21.572743 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:21.579828 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:21.580006 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:21.580209 13251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:21.590282 13251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:21.591661 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:21.593004 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:21.605846 13256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:21.605844 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:21.605846 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:21.606719 13251 server_base.cc:1034] running on GCE node
I20260925 05:07:21.607332 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:21.608227 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:21.609426 13251 hybrid_clock.cc:648] HybridClock initialized: now 1790312841609371 us; error 46 us; skew 500 ppm
I20260925 05:07:21.609738 13251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:21 dist-test-slave-wpdm krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312841, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:21.612840 13251 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:07:21.614001 13251 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:21.614144 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:21.614477 13251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:21.617448 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "01b4bfa5a1004e34bd6a71cd68fcf258"
format_stamp: "Formatted at 2026-09-25 05:07:21 on dist-test-slave-wpdm"
server_key: "dfe29fa531a1a07266cc48b5f1c78554"
server_key_iv: "5da581b7257b517d8625bc86492b6a74"
server_key_version: "encryptionkey@0"
I20260925 05:07:21.618335 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "01b4bfa5a1004e34bd6a71cd68fcf258"
format_stamp: "Formatted at 2026-09-25 05:07:21 on dist-test-slave-wpdm"
server_key: "dfe29fa531a1a07266cc48b5f1c78554"
server_key_iv: "5da581b7257b517d8625bc86492b6a74"
server_key_version: "encryptionkey@0"
I20260925 05:07:21.623364 13251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:07:21.626716 13265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:21.628010 13251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:21.628187 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "01b4bfa5a1004e34bd6a71cd68fcf258"
format_stamp: "Formatted at 2026-09-25 05:07:21 on dist-test-slave-wpdm"
server_key: "dfe29fa531a1a07266cc48b5f1c78554"
server_key_iv: "5da581b7257b517d8625bc86492b6a74"
server_key_version: "encryptionkey@0"
I20260925 05:07:21.628401 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:21.682191 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:21.686925 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:21.687242 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:21.688288 13251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:21.689707 13251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:21.689837 13251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:21.689980 13251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:21.690090 13251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:21.723845 13251 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:37489
I20260925 05:07:21.723886 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:37489 every 8 connection(s)
I20260925 05:07:21.724747 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:21.731371 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13251
I20260925 05:07:21.731606 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:21.732420 9052 external_mini_cluster.cc:1428] Setting key f5c8b58f1b8b8a584ce6629fdbedaf7e
Sep 25 05:07:21 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312841, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:21.746652 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:21.727133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:55227 (local address 127.8.215.62:45575)
0925 05:07:21.727571 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:21.727575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:21.728563 (+ 988us) server_negotiation.cc:407] Connection header received
0925 05:07:21.730314 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:21.730319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:21.730418 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:21.730538 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:21.732607 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:21.733621 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:21.735562 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:21.736400 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:21.736552 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:21.740312 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:21.740321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:21.740328 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:21.740397 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:21.743108 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:21.744057 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:21.744061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:21.744064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:21.744175 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:21.744756 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:21.744761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:21.744766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:21.745063 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:21.745169 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:21.746135 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 05:07:21.746339 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":242,"threads_started":1}
I20260925 05:07:21.748525 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:21.727650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:45575 (local address 127.8.215.1:55227)
0925 05:07:21.728390 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:21.728431 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:21.730061 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:21.730756 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:21.730764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:21.731547 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:21.732387 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:21.732416 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:21.733789 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:21.733793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:21.735409 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:21.735421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:21.736609 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:21.736614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:21.736857 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:21.737556 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:21.737578 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:21.740051 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:21.743250 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:21.743262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:21.743283 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:21.743931 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:21.744285 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:21.744293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:21.744300 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:21.744568 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:21.745180 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:21.745192 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:21.745668 (+ 476us) client_negotiation.cc:769] Sending connection context
0925 05:07:21.746028 (+ 360us) client_negotiation.cc:240] Negotiation successful
0925 05:07:21.746640 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":293,"threads_started":1}
I20260925 05:07:21.750469 13378 heartbeater.cc:347] Connected to a master server at 127.8.215.62:45575
I20260925 05:07:21.750808 13378 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:21.751497 13378 heartbeater.cc:511] Master 127.8.215.62:45575 requested a full tablet report, sending...
I20260925 05:07:21.753695 13190 ts_manager.cc:194] Registered new tserver with Master: 01b4bfa5a1004e34bd6a71cd68fcf258 (127.8.215.1:37489)
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
I20260925 05:07:21.755268 13190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:55227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:07:21.780686 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45575
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:22.104056 13384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:22.104343 13384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:22.104509 13384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:22.111866 13384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:22.111965 13384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:22.112161 13384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:22.122246 13384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:22.123510 13384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:22.124895 13384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:22.137764 13389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:22.137717 13392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:22.137678 13390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:22.138639 13384 server_base.cc:1034] running on GCE node
I20260925 05:07:22.139200 13384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:22.140096 13384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:22.141315 13384 hybrid_clock.cc:648] HybridClock initialized: now 1790312842141268 us; error 68 us; skew 500 ppm
I20260925 05:07:22.141672 13384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:22 dist-test-slave-wpdm krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312842, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:22.144920 13384 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:07:22.146023 13384 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:22.146145 13384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:22.146515 13384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:22.149502 13384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "71e031fc138a4c79be7a80def8cb3013"
format_stamp: "Formatted at 2026-09-25 05:07:22 on dist-test-slave-wpdm"
server_key: "c9fba59b6d3b177f3733d829e44b4286"
server_key_iv: "6f6ca665b73bc7044c6e5ca6d05ab249"
server_key_version: "encryptionkey@0"
I20260925 05:07:22.150285 13384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "71e031fc138a4c79be7a80def8cb3013"
format_stamp: "Formatted at 2026-09-25 05:07:22 on dist-test-slave-wpdm"
server_key: "c9fba59b6d3b177f3733d829e44b4286"
server_key_iv: "6f6ca665b73bc7044c6e5ca6d05ab249"
server_key_version: "encryptionkey@0"
I20260925 05:07:22.155483 13384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:22.158785 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:22.160034 13384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:22.160207 13384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "71e031fc138a4c79be7a80def8cb3013"
format_stamp: "Formatted at 2026-09-25 05:07:22 on dist-test-slave-wpdm"
server_key: "c9fba59b6d3b177f3733d829e44b4286"
server_key_iv: "6f6ca665b73bc7044c6e5ca6d05ab249"
server_key_version: "encryptionkey@0"
I20260925 05:07:22.160414 13384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:22.182674 13384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:22.186606 13384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:22.186880 13384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:22.187808 13384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:22.189249 13384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:22.189366 13384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:22.189502 13384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:22.189592 13384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:22.222250 13384 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:39443
I20260925 05:07:22.222397 13510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:39443 every 8 connection(s)
I20260925 05:07:22.223115 13384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:22.230017 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13384
I20260925 05:07:22.230206 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:22.230623 9052 external_mini_cluster.cc:1428] Setting key e3d18fb147113d551d19f203ce6168ac
Sep 25 05:07:22 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312842, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:22.243039 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.225593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:58377 (local address 127.8.215.62:45575)
0925 05:07:22.225759 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.225764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.226905 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 05:07:22.228673 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.228678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.228786 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.228891 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:22.230399 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.231129 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.233044 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.233483 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.233614 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.236966 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:22.236972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:22.236975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:22.237005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:22.239527 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.240243 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.240251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.240257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.240405 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.240941 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.240949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.240955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.241329 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:22.241497 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.242327 (+ 830us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.242575 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:07:22.243791 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.226088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:45575 (local address 127.8.215.2:58377)
0925 05:07:22.226747 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.226788 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.228420 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.229108 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.229116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.229599 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:22.230227 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.230245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.231263 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.231268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.232898 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.232909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.233695 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.233704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.233935 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.234615 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:22.234637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:22.236678 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:22.239650 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:22.239657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:22.239670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:22.240108 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:22.240516 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:22.240519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:22.240521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:22.240773 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:22.241516 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:22.241527 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:22.241932 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.242179 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.242576 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":272,"threads_started":1}
I20260925 05:07:22.245554 13511 heartbeater.cc:347] Connected to a master server at 127.8.215.62:45575
I20260925 05:07:22.245889 13511 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:22.246685 13511 heartbeater.cc:511] Master 127.8.215.62:45575 requested a full tablet report, sending...
I20260925 05:07:22.248217 13190 ts_manager.cc:194] Registered new tserver with Master: 71e031fc138a4c79be7a80def8cb3013 (127.8.215.2:39443)
I20260925 05:07:22.249218 13190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:58377
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:07:22.277099 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45575
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:22.597124 13516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:22.597416 13516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:22.597584 13516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:22.604878 13516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:22.604996 13516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:22.605214 13516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:22.615068 13516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:22.616446 13516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:22.617765 13516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:22.630420 13522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:22.630478 13524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:22.630504 13521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:22.631918 13516 server_base.cc:1034] running on GCE node
I20260925 05:07:22.632465 13516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:22.633455 13516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:22.634653 13516 hybrid_clock.cc:648] HybridClock initialized: now 1790312842634608 us; error 36 us; skew 500 ppm
I20260925 05:07:22.634985 13516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:22 dist-test-slave-wpdm krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312842, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:22.638077 13516 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:07:22.639178 13516 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:22.639313 13516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:22.639613 13516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:22.642634 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "2ea2f387b6e544ac8c616771ca34afbd"
format_stamp: "Formatted at 2026-09-25 05:07:22 on dist-test-slave-wpdm"
server_key: "3010f56ee05f4c5d32cb9b757cefa791"
server_key_iv: "06c099bb12be6d47e6f8dc1568267141"
server_key_version: "encryptionkey@0"
I20260925 05:07:22.643455 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "2ea2f387b6e544ac8c616771ca34afbd"
format_stamp: "Formatted at 2026-09-25 05:07:22 on dist-test-slave-wpdm"
server_key: "3010f56ee05f4c5d32cb9b757cefa791"
server_key_iv: "06c099bb12be6d47e6f8dc1568267141"
server_key_version: "encryptionkey@0"
I20260925 05:07:22.648803 13516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 05:07:22.652007 13530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:22.653350 13516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:22.653519 13516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "2ea2f387b6e544ac8c616771ca34afbd"
format_stamp: "Formatted at 2026-09-25 05:07:22 on dist-test-slave-wpdm"
server_key: "3010f56ee05f4c5d32cb9b757cefa791"
server_key_iv: "06c099bb12be6d47e6f8dc1568267141"
server_key_version: "encryptionkey@0"
I20260925 05:07:22.653729 13516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:22.676147 13516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:22.680521 13516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:22.680861 13516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:22.681838 13516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:22.683175 13516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:22.683305 13516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:22.683427 13516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:22.683526 13516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:22.715384 13516 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:42285
I20260925 05:07:22.715437 13642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:42285 every 8 connection(s)
I20260925 05:07:22.716281 13516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:22.726218 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13516
I20260925 05:07:22.726423 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:22.726922 9052 external_mini_cluster.cc:1428] Setting key 1a3adf44ca75667718e1b15f56c58dbb
Sep 25 05:07:22 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312842, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:22.736150 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.718708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:43513 (local address 127.8.215.62:45575)
0925 05:07:22.718893 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.718898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.720127 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 05:07:22.722040 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.722045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.722171 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.722284 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:22.723860 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.724590 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.726125 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.726717 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.726844 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.730827 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:22.730832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:22.730835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:22.730865 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:22.733528 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.734199 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.734202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.734205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.734317 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.734676 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.734679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.734681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.734944 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:22.735050 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.735725 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.735908 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:07:22.737141 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.719221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:45575 (local address 127.8.215.3:43513)
0925 05:07:22.719950 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.719988 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.721752 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.722520 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.722528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.723040 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:22.723679 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.723696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.724757 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.724762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.725980 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.725992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.726953 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.726961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.727209 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.727973 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:22.727999 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:22.730657 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:22.733645 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:22.733652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:22.733664 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:22.734093 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:22.734404 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:22.734408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:22.734410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:22.734581 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:22.735048 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:22.735054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:22.735380 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.735601 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.736003 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":293,"threads_started":1}
I20260925 05:07:22.738480 13643 heartbeater.cc:347] Connected to a master server at 127.8.215.62:45575
I20260925 05:07:22.738780 13643 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:22.739374 13643 heartbeater.cc:511] Master 127.8.215.62:45575 requested a full tablet report, sending...
I20260925 05:07:22.740782 13190 ts_manager.cc:194] Registered new tserver with Master: 2ea2f387b6e544ac8c616771ca34afbd (127.8.215.3:42285)
I20260925 05:07:22.741778 13190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:43513
I20260925 05:07:22.742475 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:22.758440 13378 heartbeater.cc:503] Master 127.8.215.62:45575 was elected leader, sending a full tablet report...
I20260925 05:07:22.760260 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.747828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.8.215.62:45575)
0925 05:07:22.747992 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.747996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.748527 (+ 531us) server_negotiation.cc:407] Connection header received
0925 05:07:22.748660 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.748663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.748719 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.748890 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:22.750227 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.750915 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.752070 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.752469 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.752554 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.754778 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:22.754782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:22.754786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:22.754813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:22.757220 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.757857 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.757861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.757863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.757987 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.758339 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.758343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.758346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.758751 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:22.758975 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.759788 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.760028 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:07:22.763921 13190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:22.770787 13190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:22.795089 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.782722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.8.215.2:39443)
0925 05:07:22.784974 (+ 2252us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.784985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.785019 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:07:22.785122 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.785133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.786419 (+ 1286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.786899 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:22.787876 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.789277 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.792438 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.793791 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.793944 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.794142 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.794556 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.794737 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2102,"thread_start_us":2007,"threads_started":1}
I20260925 05:07:22.795202 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.785974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:42285 (local address 127.0.0.1:51070)
0925 05:07:22.787545 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.787565 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.787679 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.788469 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.788474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.788497 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:22.788830 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.788839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.790647 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.790653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.792112 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.792122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.794284 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.794289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.794406 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.794428 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.794536 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.794720 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"mutex_wait_us":6,"thread_start_us":1280,"threads_started":1}
I20260925 05:07:22.795614 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.782614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:39443 (local address 127.0.0.1:54934)
0925 05:07:22.784075 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.784134 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.784337 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.786641 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.786648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.786702 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:22.787201 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.787222 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.789445 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.789451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.792275 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.792290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.794074 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.794086 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.794306 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.794352 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.794493 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.794722 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":186,"spinlock_wait_cycles":20096,"thread_start_us":2426,"threads_started":1}
I20260925 05:07:22.795826 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.783768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.8.215.1:37489)
0925 05:07:22.786019 (+ 2251us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.786029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.786056 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:07:22.786152 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.786160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.786400 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.787841 (+ 1441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:22.788073 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.789618 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.792457 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.793854 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.794242 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.794467 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.795280 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.795459 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":1898,"threads_started":1}
I20260925 05:07:22.795907 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.785815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.8.215.3:42285)
0925 05:07:22.787903 (+ 2088us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.787913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.787945 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:07:22.788034 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.788041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.788268 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.788461 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:22.789074 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.790513 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.792266 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.793769 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.793916 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.794169 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.795428 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.795589 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1926,"thread_start_us":1771,"threads_started":1}
I20260925 05:07:22.796834 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.783230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:36982)
0925 05:07:22.785265 (+ 2035us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.785282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.785386 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.786641 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.786648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.786695 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:22.787762 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.787773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.789766 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.789771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.792281 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.792290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.794102 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.794107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.794701 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.794729 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.794812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.794949 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1929,"mutex_wait_us":626,"spinlock_wait_cycles":896,"thread_start_us":685,"threads_started":1}
I20260925 05:07:22.799379 13578 tablet_service.cc:1467] Processing CreateTablet for tablet 3d200a98c9324bce9278fef1dd1754fb (DEFAULT_TABLE table=test-table [id=c3b21a60b13a4754846229d4b1a5888c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:22.799446 13446 tablet_service.cc:1467] Processing CreateTablet for tablet 3d200a98c9324bce9278fef1dd1754fb (DEFAULT_TABLE table=test-table [id=c3b21a60b13a4754846229d4b1a5888c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:22.800028 13313 tablet_service.cc:1467] Processing CreateTablet for tablet 3d200a98c9324bce9278fef1dd1754fb (DEFAULT_TABLE table=test-table [id=c3b21a60b13a4754846229d4b1a5888c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:22.800631 13578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d200a98c9324bce9278fef1dd1754fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:22.800792 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d200a98c9324bce9278fef1dd1754fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:22.801226 13313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d200a98c9324bce9278fef1dd1754fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:22.813115 13661 tablet_bootstrap.cc:492] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Bootstrap starting.
I20260925 05:07:22.815522 13663 tablet_bootstrap.cc:492] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: Bootstrap starting.
I20260925 05:07:22.816318 13662 tablet_bootstrap.cc:492] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Bootstrap starting.
I20260925 05:07:22.816933 13661 tablet_bootstrap.cc:654] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:22.818233 13661 log.cc:824] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:22.819993 13662 tablet_bootstrap.cc:654] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:22.821376 13661 tablet_bootstrap.cc:492] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: No bootstrap required, opened a new log
I20260925 05:07:22.821403 13662 log.cc:824] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:22.821655 13661 ts_tablet_manager.cc:1397] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 05:07:22.822335 13663 tablet_bootstrap.cc:654] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:22.823503 13663 log.cc:824] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:22.824535 13662 tablet_bootstrap.cc:492] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: No bootstrap required, opened a new log
I20260925 05:07:22.824873 13662 ts_tablet_manager.cc:1397] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 05:07:22.826756 13663 tablet_bootstrap.cc:492] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: No bootstrap required, opened a new log
I20260925 05:07:22.827039 13663 ts_tablet_manager.cc:1397] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 05:07:22.829818 13661 raft_consensus.cc:348] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.830178 13661 raft_consensus.cc:374] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:22.830338 13661 raft_consensus.cc:729] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea2f387b6e544ac8c616771ca34afbd, State: Initialized, Role: FOLLOWER
I20260925 05:07:22.830982 13661 consensus_queue.cc:260] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.833461 13643 heartbeater.cc:503] Master 127.8.215.62:45575 was elected leader, sending a full tablet report...
I20260925 05:07:22.833973 13661 ts_tablet_manager.cc:1428] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260925 05:07:22.835366 13662 raft_consensus.cc:348] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.835789 13662 raft_consensus.cc:374] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:22.835879 13662 raft_consensus.cc:729] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e031fc138a4c79be7a80def8cb3013, State: Initialized, Role: FOLLOWER
I20260925 05:07:22.836468 13662 consensus_queue.cc:260] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.837559 13511 heartbeater.cc:503] Master 127.8.215.62:45575 was elected leader, sending a full tablet report...
I20260925 05:07:22.839087 13663 raft_consensus.cc:348] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.839411 13662 ts_tablet_manager.cc:1428] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 05:07:22.839505 13663 raft_consensus.cc:374] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:22.839603 13663 raft_consensus.cc:729] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b4bfa5a1004e34bd6a71cd68fcf258, State: Initialized, Role: FOLLOWER
I20260925 05:07:22.840178 13663 consensus_queue.cc:260] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.841835 13663 ts_tablet_manager.cc:1428] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260925 05:07:22.848163 13667 raft_consensus.cc:482] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:22.848426 13667 raft_consensus.cc:504] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.850018 13667 leader_election.cc:288] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e031fc138a4c79be7a80def8cb3013 (127.8.215.2:39443), 01b4bfa5a1004e34bd6a71cd68fcf258 (127.8.215.1:37489)
I20260925 05:07:22.856171 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.850269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:58783 (local address 127.8.215.2:39443)
0925 05:07:22.850444 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.850453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.850751 (+ 298us) server_negotiation.cc:407] Connection header received
0925 05:07:22.850831 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.850838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.850916 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.851009 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:22.851442 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.852397 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.854207 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.855339 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.855443 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.855508 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.855857 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.855945 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:07:22.856266 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.850234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:39443 (local address 127.8.215.3:58783)
0925 05:07:22.850457 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.850471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.850609 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.851049 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.851053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.851077 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:22.851284 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.851310 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.852515 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.852521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.854046 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.854057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.855491 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.855497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.855687 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.855725 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.855820 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.855949 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260925 05:07:22.856983 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.850752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:50795 (local address 127.8.215.1:37489)
0925 05:07:22.850948 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.850952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.851156 (+ 204us) server_negotiation.cc:407] Connection header received
0925 05:07:22.851362 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.851370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.851447 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.851519 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:22.852077 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.852988 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.855024 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.856021 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.856147 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.856234 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.856594 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.856715 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:22.857131 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.850243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.8.215.3:50795)
0925 05:07:22.851061 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:22.851079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:22.851244 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:22.851562 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:22.851566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:22.851590 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:22.851912 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.851923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.853163 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.853168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.854868 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:22.854883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.856236 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.856242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:22.856421 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.856453 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:07:22.856569 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 05:07:22.856687 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":202,"spinlock_wait_cycles":20736,"thread_start_us":287,"threads_started":1}
I20260925 05:07:22.857432 13466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d200a98c9324bce9278fef1dd1754fb" candidate_uuid: "2ea2f387b6e544ac8c616771ca34afbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e031fc138a4c79be7a80def8cb3013" is_pre_election: true
I20260925 05:07:22.857975 13466 raft_consensus.cc:2454] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea2f387b6e544ac8c616771ca34afbd in term 0.
I20260925 05:07:22.858086 13333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d200a98c9324bce9278fef1dd1754fb" candidate_uuid: "2ea2f387b6e544ac8c616771ca34afbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" is_pre_election: true
I20260925 05:07:22.858541 13333 raft_consensus.cc:2454] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea2f387b6e544ac8c616771ca34afbd in term 0.
I20260925 05:07:22.858553 13534 leader_election.cc:302] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea2f387b6e544ac8c616771ca34afbd, 71e031fc138a4c79be7a80def8cb3013; no voters:
I20260925 05:07:22.858906 13667 raft_consensus.cc:2781] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:22.859073 13667 raft_consensus.cc:482] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:22.859205 13667 raft_consensus.cc:3037] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:22.861574 13667 raft_consensus.cc:504] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.862164 13667 leader_election.cc:288] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [CANDIDATE]: Term 1 election: Requested vote from peers 71e031fc138a4c79be7a80def8cb3013 (127.8.215.2:39443), 01b4bfa5a1004e34bd6a71cd68fcf258 (127.8.215.1:37489)
I20260925 05:07:22.862619 13466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d200a98c9324bce9278fef1dd1754fb" candidate_uuid: "2ea2f387b6e544ac8c616771ca34afbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e031fc138a4c79be7a80def8cb3013"
I20260925 05:07:22.862628 13333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d200a98c9324bce9278fef1dd1754fb" candidate_uuid: "2ea2f387b6e544ac8c616771ca34afbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258"
I20260925 05:07:22.862820 13333 raft_consensus.cc:3037] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:22.862854 13466 raft_consensus.cc:3037] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:22.866155 13466 raft_consensus.cc:2454] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea2f387b6e544ac8c616771ca34afbd in term 1.
I20260925 05:07:22.866155 13333 raft_consensus.cc:2454] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea2f387b6e544ac8c616771ca34afbd in term 1.
I20260925 05:07:22.866605 13532 leader_election.cc:302] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01b4bfa5a1004e34bd6a71cd68fcf258, 2ea2f387b6e544ac8c616771ca34afbd, 71e031fc138a4c79be7a80def8cb3013; no voters:
I20260925 05:07:22.866962 13667 raft_consensus.cc:2781] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:22.867472 13667 raft_consensus.cc:686] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 1 LEADER]: Becoming Leader. State: Replica: 2ea2f387b6e544ac8c616771ca34afbd, State: Running, Role: LEADER
I20260925 05:07:22.867950 13667 consensus_queue.cc:237] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } }
I20260925 05:07:22.872681 13190 catalog_manager.cc:5495] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ea2f387b6e544ac8c616771ca34afbd (127.8.215.3). New cstate: current_term: 1 leader_uuid: "2ea2f387b6e544ac8c616771ca34afbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea2f387b6e544ac8c616771ca34afbd" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 42285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:22.891672 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.886252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.8.215.3:42285)
0925 05:07:22.886469 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.886475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.886576 (+ 101us) server_negotiation.cc:407] Connection header received
0925 05:07:22.886808 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.886811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.886894 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.887049 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:22.887419 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.888132 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.889520 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.889929 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.890067 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.890538 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:22.891116 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:22.891275 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.891330 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.891406 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 05:07:22.899253 13466 raft_consensus.cc:1261] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 1 FOLLOWER]: Refusing update from remote peer 2ea2f387b6e544ac8c616771ca34afbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:22.899255 13333 raft_consensus.cc:1261] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 1 FOLLOWER]: Refusing update from remote peer 2ea2f387b6e544ac8c616771ca34afbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:22.900189 13671 consensus_queue.cc:1035] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e031fc138a4c79be7a80def8cb3013" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 39443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:22.900525 13667 consensus_queue.cc:1035] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b4bfa5a1004e34bd6a71cd68fcf258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:22.912360 13676 mvcc.cc:205] Tried to move back new op lower bound from 7333121404506337280 to 7333121404391268352. Current Snapshot: MvccSnapshot[applied={T|T < 7333121404506337280}]
I20260925 05:07:22.924810 13190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:22.925077 13190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:22.928411 13190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72ac9f1dac7f4bfa9857b7d3306191d8: Sending DeleteTablet for 3 replicas of tablet 3d200a98c9324bce9278fef1dd1754fb
I20260925 05:07:22.929942 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d200a98c9324bce9278fef1dd1754fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:22 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:54934
I20260925 05:07:22.929980 13313 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d200a98c9324bce9278fef1dd1754fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:22 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:36982
I20260925 05:07:22.930018 13578 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d200a98c9324bce9278fef1dd1754fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:22 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:51070
I20260925 05:07:22.931404 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13251
I20260925 05:07:22.942402 13686 tablet_replica.cc:330] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: stopping tablet replica
I20260925 05:07:22.943591 13686 raft_consensus.cc:2229] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:22.944229 13686 raft_consensus.cc:2258] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:22.947525 13686 ts_tablet_manager.cc:1905] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:22.949622 13684 tablet_replica.cc:330] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: stopping tablet replica
I20260925 05:07:22.950085 13684 raft_consensus.cc:2229] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:22.950829 13684 raft_consensus.cc:2258] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 05:07:22 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:22.957382 13685 tablet_replica.cc:330] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258: stopping tablet replica
I20260925 05:07:22.959270 13684 ts_tablet_manager.cc:1905] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:22.959750 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:22.933238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.8.215.1:37489)
0925 05:07:22.933467 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:22.933477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:22.938678 (+ 5201us) server_negotiation.cc:407] Connection header received
0925 05:07:22.939539 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:22.939547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:22.939658 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:22.939741 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:22.942771 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.943933 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.946116 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:22.946780 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:22.946893 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:22.952415 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:22.952431 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:22.952449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:22.952495 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:22.956466 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.957379 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.957390 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.957393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.957554 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:22.958003 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:22.958009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:22.958012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:22.958493 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:22.958781 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:22.959143 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 05:07:22.959382 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:07:22.960417 13373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:22.966856 13684 ts_tablet_manager.cc:1918] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:22.967026 13684 log.cc:1196] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/3d200a98c9324bce9278fef1dd1754fb
I20260925 05:07:22.967552 13684 ts_tablet_manager.cc:1939] T 3d200a98c9324bce9278fef1dd1754fb P 71e031fc138a4c79be7a80def8cb3013: Deleting consensus metadata
I20260925 05:07:22.969542 13177 catalog_manager.cc:4841] TS 71e031fc138a4c79be7a80def8cb3013 (127.8.215.2:39443): tablet 3d200a98c9324bce9278fef1dd1754fb (table test-table [id=c3b21a60b13a4754846229d4b1a5888c]) successfully deleted
I20260925 05:07:22.969580 13686 ts_tablet_manager.cc:1918] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:22.969859 13686 log.cc:1196] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/3d200a98c9324bce9278fef1dd1754fb
I20260925 05:07:22.970531 13686 ts_tablet_manager.cc:1939] T 3d200a98c9324bce9278fef1dd1754fb P 2ea2f387b6e544ac8c616771ca34afbd: Deleting consensus metadata
I20260925 05:07:22.972242 13177 catalog_manager.cc:4841] TS 2ea2f387b6e544ac8c616771ca34afbd (127.8.215.3:42285): tablet 3d200a98c9324bce9278fef1dd1754fb (table test-table [id=c3b21a60b13a4754846229d4b1a5888c]) successfully deleted
I20260925 05:07:22.957866 13685 raft_consensus.cc:2229] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:23.049914 13685 raft_consensus.cc:2258] T 3d200a98c9324bce9278fef1dd1754fb P 01b4bfa5a1004e34bd6a71cd68fcf258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:23.050069 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13251
I20260925 05:07:23.073316 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13384
W20260925 05:07:23.073315 13175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.215.1:37489 (error 108)
I20260925 05:07:23.076642 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.076250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:36990)
0925 05:07:23.076406 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.076512 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:07:23.076974 13175 catalog_manager.cc:4584] TS 01b4bfa5a1004e34bd6a71cd68fcf258 (127.8.215.1:37489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d200a98c9324bce9278fef1dd1754fb: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Sep 25 05:07:23 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:07:23.090327 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.074580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.8.215.2:39443)
0925 05:07:23.074811 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:23.074819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:23.074859 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:07:23.074940 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:23.074948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:23.075079 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:23.075214 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:23.076800 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:23.077936 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:23.079138 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:23.079780 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:23.079922 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:23.083872 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:23.083889 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:23.083909 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:23.083953 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:23.087519 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:23.088149 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:23.088159 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:23.088163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:23.088316 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:23.088651 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:23.088657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:23.088660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:23.089266 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:23.089555 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:23.089740 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:07:23.089946 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:07:23.091003 13506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:23.130409 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.129975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:36998)
0925 05:07:23.130168 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.130256 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 05:07:23.131428 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.131102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:37014)
0925 05:07:23.131242 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.131322 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 05:07:23.192206 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.191663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:37018)
0925 05:07:23.191925 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.192026 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 05:07:23.193385 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.192969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:37030)
0925 05:07:23.193142 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.193229 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 05:07:23.208389 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13384
I20260925 05:07:23.225286 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13516
Sep 25 05:07:23 dist-test-slave-wpdm krb5kdc[13146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:07:23.243115 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.226242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.8.215.3:42285)
0925 05:07:23.226422 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:23.226430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:23.226452 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:07:23.226566 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:23.226570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:23.226630 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:23.226711 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:23.228396 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:23.229390 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:23.231221 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:23.231647 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:23.231736 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:23.236959 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:23.236981 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:23.237001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:23.237041 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:23.239988 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:23.240766 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:23.240777 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:23.240781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:23.240954 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:23.241346 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:23.241351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:23.241354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:23.241782 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:23.242095 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:23.242388 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:07:23.242624 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:07:23.243892 13638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:23.274012 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.273571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:37042)
0925 05:07:23.273754 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.273843 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 05:07:23.275084 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.274695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:37058)
0925 05:07:23.274900 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.274975 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 05:07:23.363107 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13516
I20260925 05:07:23.382033 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13159
I20260925 05:07:23.383098 13220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:23.489238 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.488649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37489 (local address 127.0.0.1:37074)
0925 05:07:23.488902 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:23.489021 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:07:23.489254 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13159
2026-09-25T05:07:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4109 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T05:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:23Z Disabled control of system clock
I20260925 05:07:23.556071 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40025
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:07:23.874094 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:23.874377 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:23.874534 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:23.881520 13696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:23.881637 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:23.881718 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:23.881842 13696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:23.881902 13696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:23.891407 13696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:40025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:23.892858 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:23.895030 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:23.907150 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:23.907191 13704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:23.907240 13702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:23.908015 13696 server_base.cc:1034] running on GCE node
I20260925 05:07:23.908838 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:23.910156 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:23.911410 13696 hybrid_clock.cc:648] HybridClock initialized: now 1790312843911351 us; error 41 us; skew 500 ppm
I20260925 05:07:23.911767 13696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:23.913971 13696 webserver.cc:460] Webserver started at http://127.8.215.62:35857/ using document root <none> and password file <none>
I20260925 05:07:23.914669 13696 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:23.914778 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:23.915107 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:23.917922 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "64ee5c9c427a47daaa9d2f49d25b0528"
format_stamp: "Formatted at 2026-09-25 05:07:23 on dist-test-slave-wpdm"
server_key: "50b6a7133843f2b7e41b6cd264aec111"
server_key_iv: "bcba28ec0c5f083e43457e5122e70b97"
server_key_version: "encryptionkey@0"
I20260925 05:07:23.918677 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "64ee5c9c427a47daaa9d2f49d25b0528"
format_stamp: "Formatted at 2026-09-25 05:07:23 on dist-test-slave-wpdm"
server_key: "50b6a7133843f2b7e41b6cd264aec111"
server_key_iv: "bcba28ec0c5f083e43457e5122e70b97"
server_key_version: "encryptionkey@0"
I20260925 05:07:23.923436 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 05:07:23.926659 13710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:23.927911 13696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:07:23.928083 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "64ee5c9c427a47daaa9d2f49d25b0528"
format_stamp: "Formatted at 2026-09-25 05:07:23 on dist-test-slave-wpdm"
server_key: "50b6a7133843f2b7e41b6cd264aec111"
server_key_iv: "bcba28ec0c5f083e43457e5122e70b97"
server_key_version: "encryptionkey@0"
I20260925 05:07:23.928287 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:23.933811 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:23.934535 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:23.934850 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:23.952417 13696 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:40025
I20260925 05:07:23.952453 13761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:40025 every 8 connection(s)
I20260925 05:07:23.953747 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:23.957267 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:23.962654 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13696
I20260925 05:07:23.962849 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:23.963456 9052 external_mini_cluster.cc:1428] Setting key 7a9c8d391269d89dce3146f84e84eb3b
I20260925 05:07:23.968322 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528: Bootstrap starting.
I20260925 05:07:23.972199 13762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:23.972857 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:23.965564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.8.215.62:40025)
0925 05:07:23.966586 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:23.966599 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:23.966646 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:07:23.968882 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:23.968909 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:23.969394 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:23.969820 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:23.970867 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:23.970875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:23.970891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:23.970907 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:23.970982 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:23.971003 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:23.971210 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:23.971333 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 05:07:23.971592 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":310,"threads_started":1}
I20260925 05:07:23.973462 13762 log.cc:824] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:23.976212 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528: No bootstrap required, opened a new log
I20260925 05:07:23.983060 13762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40025 } }
I20260925 05:07:23.983428 13762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:23.983578 13762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ee5c9c427a47daaa9d2f49d25b0528, State: Initialized, Role: FOLLOWER
I20260925 05:07:23.984130 13762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40025 } }
I20260925 05:07:23.984414 13762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:23.984555 13762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:23.984706 13762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:23.986809 13762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40025 } }
I20260925 05:07:23.987246 13762 leader_election.cc:302] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64ee5c9c427a47daaa9d2f49d25b0528; no voters:
I20260925 05:07:23.987736 13762 leader_election.cc:288] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:23.987890 13767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:23.988648 13767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [term 1 LEADER]: Becoming Leader. State: Replica: 64ee5c9c427a47daaa9d2f49d25b0528, State: Running, Role: LEADER
I20260925 05:07:23.989230 13767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40025 } }
I20260925 05:07:23.989799 13762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:23.992198 13768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40025 } } }
I20260925 05:07:23.992564 13768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:23.992249 13769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64ee5c9c427a47daaa9d2f49d25b0528. Latest consensus state: current_term: 1 leader_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ee5c9c427a47daaa9d2f49d25b0528" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40025 } } }
I20260925 05:07:23.992700 13769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:23.994993 13774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:23.999255 13774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:24.007290 13774 catalog_manager.cc:1347] Generated new cluster ID: 06ae1703ddc340de816fb35cdd605ab1
I20260925 05:07:24.007428 13774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:24.017431 13774 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:24.017814 13774 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:24.034703 13774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528: Generated new TSK 0
I20260925 05:07:24.035578 13774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:07:24.044287 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40025
--builtin_ntp_servers=127.8.215.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:07:24.363060 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:24.363320 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:24.363492 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:24.370448 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:24.370564 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:24.370743 13786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:24.379678 13786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:24.380955 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:24.382699 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:24.395311 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:24.395289 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:24.395454 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:24.396811 13786 server_base.cc:1034] running on GCE node
I20260925 05:07:24.397400 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:24.398200 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:24.399397 13786 hybrid_clock.cc:648] HybridClock initialized: now 1790312844399362 us; error 26 us; skew 500 ppm
I20260925 05:07:24.399701 13786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:24.401871 13786 webserver.cc:460] Webserver started at http://127.8.215.1:43627/ using document root <none> and password file <none>
I20260925 05:07:24.402521 13786 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:24.402629 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:24.402942 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:24.405630 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "2168920747984e4288d718dabe44a4aa"
format_stamp: "Formatted at 2026-09-25 05:07:24 on dist-test-slave-wpdm"
server_key: "dd3d9e030d42ab25c3ba15512a35bac2"
server_key_iv: "8c478aa4dcaa7a07d4d4f2920becb59f"
server_key_version: "encryptionkey@0"
I20260925 05:07:24.406360 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "2168920747984e4288d718dabe44a4aa"
format_stamp: "Formatted at 2026-09-25 05:07:24 on dist-test-slave-wpdm"
server_key: "dd3d9e030d42ab25c3ba15512a35bac2"
server_key_iv: "8c478aa4dcaa7a07d4d4f2920becb59f"
server_key_version: "encryptionkey@0"
I20260925 05:07:24.410986 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:07:24.414129 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:24.415320 13786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:07:24.415496 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "2168920747984e4288d718dabe44a4aa"
format_stamp: "Formatted at 2026-09-25 05:07:24 on dist-test-slave-wpdm"
server_key: "dd3d9e030d42ab25c3ba15512a35bac2"
server_key_iv: "8c478aa4dcaa7a07d4d4f2920becb59f"
server_key_version: "encryptionkey@0"
I20260925 05:07:24.415709 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:24.421099 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:24.421828 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:24.422097 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:24.422966 13786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:24.424273 13786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:24.424392 13786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:24.424527 13786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:24.424613 13786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:24.456902 13786 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:40717
I20260925 05:07:24.456939 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:40717 every 8 connection(s)
I20260925 05:07:24.458201 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:24.463104 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13786
I20260925 05:07:24.463299 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:24.464100 9052 external_mini_cluster.cc:1428] Setting key f717b4292768810fe9903f7b001f90e8
I20260925 05:07:24.467751 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40025
--builtin_ntp_servers=127.8.215.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 05:07:24.475971 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:24.460767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:34081 (local address 127.8.215.62:40025)
0925 05:07:24.460942 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:24.460946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:24.462225 (+ 1279us) server_negotiation.cc:407] Connection header received
0925 05:07:24.464591 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:24.464595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:24.464646 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:24.464747 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:24.475253 (+ 10506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:24.475257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:24.475260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:24.475278 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:24.475311 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:24.475323 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:24.475454 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:24.475652 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 05:07:24.475721 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:07:24.477077 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:24.461209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40025 (local address 127.8.215.1:34081)
0925 05:07:24.462076 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:24.462111 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:24.464303 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:24.465343 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:24.465355 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:24.465482 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:24.474898 (+ 9416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:24.474901 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:24.474950 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:24.475050 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:24.475421 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:24.475435 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:24.475444 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:07:24.475739 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 05:07:24.476032 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":248,"threads_started":1}
I20260925 05:07:24.478858 13913 heartbeater.cc:347] Connected to a master server at 127.8.215.62:40025
I20260925 05:07:24.479153 13913 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:24.479732 13913 heartbeater.cc:511] Master 127.8.215.62:40025 requested a full tablet report, sending...
I20260925 05:07:24.482089 13727 ts_manager.cc:194] Registered new tserver with Master: 2168920747984e4288d718dabe44a4aa (127.8.215.1:40717)
W20260925 05:07:24.816335 13916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:24.816622 13916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:24.816764 13916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:24.824347 13916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:24.824462 13916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:24.824581 13916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:24.833923 13916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:24.835168 13916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:24.836937 13916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:24.850339 13922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:24.850541 13921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:24.850695 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:24.851643 13916 server_base.cc:1034] running on GCE node
I20260925 05:07:24.852378 13916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:24.853432 13916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:24.854686 13916 hybrid_clock.cc:648] HybridClock initialized: now 1790312844854621 us; error 46 us; skew 500 ppm
I20260925 05:07:24.855154 13916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:24.857517 13916 webserver.cc:460] Webserver started at http://127.8.215.2:42703/ using document root <none> and password file <none>
I20260925 05:07:24.858238 13916 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:24.858351 13916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:24.858671 13916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:24.861626 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "cf248985c8204b84a6ea4679c4334b12"
format_stamp: "Formatted at 2026-09-25 05:07:24 on dist-test-slave-wpdm"
server_key: "251ed52050b2126e50bf4ba32ac726ef"
server_key_iv: "0f22363ac290b98db2215cefce076cd7"
server_key_version: "encryptionkey@0"
I20260925 05:07:24.862385 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "cf248985c8204b84a6ea4679c4334b12"
format_stamp: "Formatted at 2026-09-25 05:07:24 on dist-test-slave-wpdm"
server_key: "251ed52050b2126e50bf4ba32ac726ef"
server_key_iv: "0f22363ac290b98db2215cefce076cd7"
server_key_version: "encryptionkey@0"
I20260925 05:07:24.867390 13916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:07:24.870631 13930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:24.871860 13916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 05:07:24.872036 13916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "cf248985c8204b84a6ea4679c4334b12"
format_stamp: "Formatted at 2026-09-25 05:07:24 on dist-test-slave-wpdm"
server_key: "251ed52050b2126e50bf4ba32ac726ef"
server_key_iv: "0f22363ac290b98db2215cefce076cd7"
server_key_version: "encryptionkey@0"
I20260925 05:07:24.872270 13916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:24.877686 13916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:24.878435 13916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:24.878695 13916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:24.879544 13916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:24.880918 13916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:24.881033 13916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:24.881166 13916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:24.881258 13916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:24.913077 13916 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:33185
I20260925 05:07:24.913156 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:33185 every 8 connection(s)
I20260925 05:07:24.914362 13916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:24.920042 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 13916
I20260925 05:07:24.920199 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:24.920761 9052 external_mini_cluster.cc:1428] Setting key 0f34ff0a7a9838447a95618900ed0cc5
I20260925 05:07:24.923290 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:24.916952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:46709 (local address 127.8.215.62:40025)
0925 05:07:24.917132 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:24.917136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:24.918848 (+ 1712us) server_negotiation.cc:407] Connection header received
0925 05:07:24.921002 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:24.921007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:24.921061 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:24.921162 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:24.922533 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:24.922538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:24.922542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:24.922560 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:24.922600 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:24.922615 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:24.922721 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:24.922956 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 05:07:24.923020 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:07:24.923799 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40025
--builtin_ntp_servers=127.8.215.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 05:07:24.924206 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:24.917740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40025 (local address 127.8.215.2:46709)
0925 05:07:24.918688 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:24.918730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:24.920636 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:24.921429 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:24.921438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:24.921539 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:24.922193 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:24.922196 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:24.922243 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:24.922346 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:24.922721 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:24.922733 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:24.922742 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:07:24.922983 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 05:07:24.923178 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
I20260925 05:07:24.925539 14043 heartbeater.cc:347] Connected to a master server at 127.8.215.62:40025
I20260925 05:07:24.925832 14043 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:24.926374 14043 heartbeater.cc:511] Master 127.8.215.62:40025 requested a full tablet report, sending...
I20260925 05:07:24.927702 13727 ts_manager.cc:194] Registered new tserver with Master: cf248985c8204b84a6ea4679c4334b12 (127.8.215.2:33185)
W20260925 05:07:25.263443 14046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:25.263718 14046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:25.263805 14046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:25.270861 14046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:25.270964 14046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:25.271075 14046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:25.280125 14046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:25.281371 14046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:25.283047 14046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:25.296044 14051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:25.296002 14052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:25.296512 14054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:25.296980 14046 server_base.cc:1034] running on GCE node
I20260925 05:07:25.297557 14046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:25.298518 14046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:25.299744 14046 hybrid_clock.cc:648] HybridClock initialized: now 1790312845299706 us; error 53 us; skew 500 ppm
I20260925 05:07:25.300069 14046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:25.302301 14046 webserver.cc:460] Webserver started at http://127.8.215.3:38525/ using document root <none> and password file <none>
I20260925 05:07:25.302980 14046 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:25.303120 14046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:25.303436 14046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:25.306339 14046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "e34ba42676f7422dba2b87c23335767c"
format_stamp: "Formatted at 2026-09-25 05:07:25 on dist-test-slave-wpdm"
server_key: "55b096b0c20ca1aee9b2580012c5051c"
server_key_iv: "8706eb24f816fb8971afeeea28d82c7f"
server_key_version: "encryptionkey@0"
I20260925 05:07:25.307103 14046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "e34ba42676f7422dba2b87c23335767c"
format_stamp: "Formatted at 2026-09-25 05:07:25 on dist-test-slave-wpdm"
server_key: "55b096b0c20ca1aee9b2580012c5051c"
server_key_iv: "8706eb24f816fb8971afeeea28d82c7f"
server_key_version: "encryptionkey@0"
I20260925 05:07:25.311998 14046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:07:25.315517 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:25.316882 14046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:25.317054 14046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "e34ba42676f7422dba2b87c23335767c"
format_stamp: "Formatted at 2026-09-25 05:07:25 on dist-test-slave-wpdm"
server_key: "55b096b0c20ca1aee9b2580012c5051c"
server_key_iv: "8706eb24f816fb8971afeeea28d82c7f"
server_key_version: "encryptionkey@0"
I20260925 05:07:25.317283 14046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:25.322897 14046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:25.323683 14046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:25.323956 14046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:25.324919 14046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:25.326269 14046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:25.326390 14046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:25.326526 14046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:25.326617 14046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:25.359226 14046 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:34091
I20260925 05:07:25.359274 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:34091 every 8 connection(s)
I20260925 05:07:25.360566 14046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:25.365105 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14046
I20260925 05:07:25.365271 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:25.365705 9052 external_mini_cluster.cc:1428] Setting key 7f9abc9ae8268b84c398722a38ef2f36
I20260925 05:07:25.368556 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.363033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:43137 (local address 127.8.215.62:40025)
0925 05:07:25.363253 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.363260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.364556 (+ 1296us) server_negotiation.cc:407] Connection header received
0925 05:07:25.366443 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.366449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.366520 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.366634 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.367880 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.367885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.367888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.367902 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.367930 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.367940 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.368057 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.368247 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.368309 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:07:25.369632 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.363530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40025 (local address 127.8.215.3:43137)
0925 05:07:25.364410 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.364445 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:25.366167 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:25.366860 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:25.366868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:25.366959 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:25.367569 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:25.367572 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:25.367607 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:25.367690 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:25.368033 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:25.368042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:25.368050 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:07:25.368338 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 05:07:25.368591 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260925 05:07:25.370962 14173 heartbeater.cc:347] Connected to a master server at 127.8.215.62:40025
I20260925 05:07:25.371271 14173 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:25.372071 14173 heartbeater.cc:511] Master 127.8.215.62:40025 requested a full tablet report, sending...
I20260925 05:07:25.373528 13727 ts_manager.cc:194] Registered new tserver with Master: e34ba42676f7422dba2b87c23335767c (127.8.215.3:34091)
I20260925 05:07:25.380843 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:25.387470 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.385151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.8.215.62:40025)
0925 05:07:25.385298 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.385302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.385786 (+ 484us) server_negotiation.cc:407] Connection header received
0925 05:07:25.385907 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.385911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.385957 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.386047 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.386962 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.386965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.386968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.386980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.387011 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.387021 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.387104 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.387141 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.387181 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:07:25.391296 13727 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:25.398077 13727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:25.428386 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.409064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:33185 (local address 127.0.0.1:47806)
0925 05:07:25.409838 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.409892 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:25.410055 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:25.426077 (+ 16022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:25.426086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:25.426207 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:25.427112 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:25.427116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:25.427146 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:25.427231 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:25.427641 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:25.427653 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:25.427660 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:07:25.427825 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 05:07:25.427908 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":536,"threads_started":1}
I20260925 05:07:25.428973 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.409221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.8.215.2:33185)
0925 05:07:25.425422 (+ 16201us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.425436 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.425475 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:07:25.425565 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.425573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.425837 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.426396 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.427372 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.427384 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.427403 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.427427 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.427479 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.427498 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.428481 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.428552 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.428651 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16045,"thread_start_us":15926,"threads_started":1}
I20260925 05:07:25.429595 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.409064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36772)
0925 05:07:25.421865 (+ 12801us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.421899 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:25.422030 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:25.426212 (+ 4182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:25.426219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:25.426238 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:25.427430 (+ 1192us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:25.427434 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:25.427447 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:25.427482 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:25.429003 (+ 1521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:25.429008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:25.429013 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:07:25.429143 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 05:07:25.429186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12297,"mutex_wait_us":511,"thread_start_us":16386,"threads_started":1}
I20260925 05:07:25.429890 14187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.409470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.8.215.1:40717)
0925 05:07:25.425722 (+ 16252us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.425730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.425753 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:07:25.425814 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.425820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.426048 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.427588 (+ 1540us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.428717 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.428766 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.428782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.428814 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.428875 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.428893 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.429230 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.429306 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.429397 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16110,"thread_start_us":16129,"threads_started":1}
I20260925 05:07:25.430663 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.410546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:34091 (local address 127.0.0.1:39338)
0925 05:07:25.426239 (+ 15693us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.426256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:25.426367 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:25.427502 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:25.427507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:25.427529 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:25.428174 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:25.428179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:25.428198 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:25.428249 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:25.430085 (+ 1836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:25.430089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:25.430094 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:07:25.430224 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 05:07:25.430268 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15563,"thread_start_us":10983,"threads_started":1}
I20260925 05:07:25.430945 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.411123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.8.215.3:34091)
0925 05:07:25.425546 (+ 14423us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.425558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.426491 (+ 933us) server_negotiation.cc:407] Connection header received
0925 05:07:25.426570 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.426577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.427347 (+ 770us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.427514 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.428851 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.428858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.428870 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.428886 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.428946 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.428968 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.429136 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.430446 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.430537 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14212,"thread_start_us":14043,"threads_started":1}
I20260925 05:07:25.433102 13978 tablet_service.cc:1467] Processing CreateTablet for tablet 5491b397f75647fa9f241703859c4070 (DEFAULT_TABLE table=test-table [id=cbc3675865dd4c4bb0ac5a17ffa84834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:25.433651 13848 tablet_service.cc:1467] Processing CreateTablet for tablet 5491b397f75647fa9f241703859c4070 (DEFAULT_TABLE table=test-table [id=cbc3675865dd4c4bb0ac5a17ffa84834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:25.434165 14108 tablet_service.cc:1467] Processing CreateTablet for tablet 5491b397f75647fa9f241703859c4070 (DEFAULT_TABLE table=test-table [id=cbc3675865dd4c4bb0ac5a17ffa84834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:25.434623 13978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5491b397f75647fa9f241703859c4070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:25.434954 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5491b397f75647fa9f241703859c4070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:25.435300 14108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5491b397f75647fa9f241703859c4070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:25.446269 14191 tablet_bootstrap.cc:492] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Bootstrap starting.
I20260925 05:07:25.449461 14192 tablet_bootstrap.cc:492] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Bootstrap starting.
I20260925 05:07:25.451125 14191 tablet_bootstrap.cc:654] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:25.452358 14191 log.cc:824] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:25.453001 14192 tablet_bootstrap.cc:654] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:25.453764 14193 tablet_bootstrap.cc:492] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Bootstrap starting.
I20260925 05:07:25.454221 14192 log.cc:824] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:25.455766 14191 tablet_bootstrap.cc:492] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: No bootstrap required, opened a new log
I20260925 05:07:25.456065 14191 ts_tablet_manager.cc:1397] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 05:07:25.458577 14193 tablet_bootstrap.cc:654] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:25.460060 14193 log.cc:824] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:25.464088 14193 tablet_bootstrap.cc:492] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: No bootstrap required, opened a new log
I20260925 05:07:25.464359 14193 ts_tablet_manager.cc:1397] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 05:07:25.464538 14192 tablet_bootstrap.cc:492] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: No bootstrap required, opened a new log
I20260925 05:07:25.464816 14192 ts_tablet_manager.cc:1397] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260925 05:07:25.466741 14191 raft_consensus.cc:348] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.467274 14191 raft_consensus.cc:374] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:25.467402 14191 raft_consensus.cc:729] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34ba42676f7422dba2b87c23335767c, State: Initialized, Role: FOLLOWER
I20260925 05:07:25.468143 14191 consensus_queue.cc:260] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.475520 14173 heartbeater.cc:503] Master 127.8.215.62:40025 was elected leader, sending a full tablet report...
I20260925 05:07:25.475558 14192 raft_consensus.cc:348] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.475551 14193 raft_consensus.cc:348] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.475984 14192 raft_consensus.cc:374] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:25.476013 14193 raft_consensus.cc:374] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:25.476096 14193 raft_consensus.cc:729] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2168920747984e4288d718dabe44a4aa, State: Initialized, Role: FOLLOWER
I20260925 05:07:25.476096 14192 raft_consensus.cc:729] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf248985c8204b84a6ea4679c4334b12, State: Initialized, Role: FOLLOWER
I20260925 05:07:25.476851 14191 ts_tablet_manager.cc:1428] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260925 05:07:25.476820 14192 consensus_queue.cc:260] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.477263 14193 consensus_queue.cc:260] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.477953 14043 heartbeater.cc:503] Master 127.8.215.62:40025 was elected leader, sending a full tablet report...
I20260925 05:07:25.479007 14192 ts_tablet_manager.cc:1428] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 05:07:25.479981 13913 heartbeater.cc:503] Master 127.8.215.62:40025 was elected leader, sending a full tablet report...
I20260925 05:07:25.481408 14193 ts_tablet_manager.cc:1428] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260925 05:07:25.612293 14174 tablet.cc:2369] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:07:25.666322 14044 tablet.cc:2369] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:25.683570 14198 raft_consensus.cc:482] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:25.683923 14198 raft_consensus.cc:504] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.685634 14198 leader_election.cc:288] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2168920747984e4288d718dabe44a4aa (127.8.215.1:40717), e34ba42676f7422dba2b87c23335767c (127.8.215.3:34091)
I20260925 05:07:25.688997 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.686019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:58481 (local address 127.8.215.3:34091)
0925 05:07:25.686239 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.686248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.687015 (+ 767us) server_negotiation.cc:407] Connection header received
0925 05:07:25.687201 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.687207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.687264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.687368 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.688305 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.688311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.688315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.688331 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.688372 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.688386 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.688545 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.688610 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.688670 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 05:07:25.689203 14187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.686217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:52019 (local address 127.8.215.1:40717)
0925 05:07:25.686470 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.686480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.687177 (+ 697us) server_negotiation.cc:407] Connection header received
0925 05:07:25.687347 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.687351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.687399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.687495 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.688521 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.688525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.688528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.688543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.688569 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.688579 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.688694 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.688827 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.688896 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:07:25.689539 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.686010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:34091 (local address 127.8.215.2:58481)
0925 05:07:25.686901 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.686917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:25.687074 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:25.687401 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:25.687408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:25.687432 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:25.688129 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:25.688131 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:25.688143 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:25.688177 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:25.688507 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:25.688511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:25.688516 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:07:25.688643 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 05:07:25.688695 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":565,"threads_started":1}
I20260925 05:07:25.689642 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.686075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.8.215.2:52019)
0925 05:07:25.687070 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.687085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:25.687212 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:25.687521 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:25.687527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:25.687557 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:25.688321 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:07:25.688324 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:25.688336 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:07:25.688368 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:25.688670 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:25.688674 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:25.688678 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:07:25.688865 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 05:07:25.688909 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":657,"threads_started":1}
I20260925 05:07:25.690716 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491b397f75647fa9f241703859c4070" candidate_uuid: "cf248985c8204b84a6ea4679c4334b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34ba42676f7422dba2b87c23335767c" is_pre_election: true
I20260925 05:07:25.690713 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491b397f75647fa9f241703859c4070" candidate_uuid: "cf248985c8204b84a6ea4679c4334b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2168920747984e4288d718dabe44a4aa" is_pre_election: true
I20260925 05:07:25.691283 13868 raft_consensus.cc:2454] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf248985c8204b84a6ea4679c4334b12 in term 0.
I20260925 05:07:25.691283 14128 raft_consensus.cc:2454] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf248985c8204b84a6ea4679c4334b12 in term 0.
I20260925 05:07:25.691964 13934 leader_election.cc:302] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf248985c8204b84a6ea4679c4334b12, e34ba42676f7422dba2b87c23335767c; no voters:
I20260925 05:07:25.692334 14198 raft_consensus.cc:2781] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:25.692476 14198 raft_consensus.cc:482] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:25.692571 14198 raft_consensus.cc:3037] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:25.694754 14198 raft_consensus.cc:504] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.695363 14198 leader_election.cc:288] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [CANDIDATE]: Term 1 election: Requested vote from peers 2168920747984e4288d718dabe44a4aa (127.8.215.1:40717), e34ba42676f7422dba2b87c23335767c (127.8.215.3:34091)
I20260925 05:07:25.695745 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491b397f75647fa9f241703859c4070" candidate_uuid: "cf248985c8204b84a6ea4679c4334b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2168920747984e4288d718dabe44a4aa"
I20260925 05:07:25.695843 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491b397f75647fa9f241703859c4070" candidate_uuid: "cf248985c8204b84a6ea4679c4334b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34ba42676f7422dba2b87c23335767c"
I20260925 05:07:25.695958 13868 raft_consensus.cc:3037] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:25.696092 14128 raft_consensus.cc:3037] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:25.698176 13868 raft_consensus.cc:2454] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf248985c8204b84a6ea4679c4334b12 in term 1.
I20260925 05:07:25.698313 14128 raft_consensus.cc:2454] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf248985c8204b84a6ea4679c4334b12 in term 1.
I20260925 05:07:25.698606 13933 leader_election.cc:302] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2168920747984e4288d718dabe44a4aa, cf248985c8204b84a6ea4679c4334b12; no voters:
I20260925 05:07:25.698928 14198 raft_consensus.cc:2781] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:25.699335 14198 raft_consensus.cc:686] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 1 LEADER]: Becoming Leader. State: Replica: cf248985c8204b84a6ea4679c4334b12, State: Running, Role: LEADER
I20260925 05:07:25.699785 14198 consensus_queue.cc:237] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } }
I20260925 05:07:25.704704 13727 catalog_manager.cc:5495] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf248985c8204b84a6ea4679c4334b12 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "cf248985c8204b84a6ea4679c4334b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf248985c8204b84a6ea4679c4334b12" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 } health_report { overall_health: UNKNOWN } } }
W20260925 05:07:25.710453 13914 tablet.cc:2369] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:25.726411 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.724259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.8.215.2:33185)
0925 05:07:25.724422 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.724426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.724445 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:07:25.724613 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.724617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.724695 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.724842 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.725923 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.725926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.725929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.725941 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.725965 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.725975 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.726062 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.726167 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.726212 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:07:25.734246 14128 raft_consensus.cc:1261] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 1 FOLLOWER]: Refusing update from remote peer cf248985c8204b84a6ea4679c4334b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:25.734305 13868 raft_consensus.cc:1261] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 1 FOLLOWER]: Refusing update from remote peer cf248985c8204b84a6ea4679c4334b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:25.735172 14202 consensus_queue.cc:1035] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34ba42676f7422dba2b87c23335767c" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:25.735472 14198 consensus_queue.cc:1035] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2168920747984e4288d718dabe44a4aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:25.746613 14205 mvcc.cc:205] Tried to move back new op lower bound from 7333121416118370304 to 7333121415990349824. Current Snapshot: MvccSnapshot[applied={T|T < 7333121416118370304}]
I20260925 05:07:25.760571 13726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:25.760892 13726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:25.764508 13726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64ee5c9c427a47daaa9d2f49d25b0528: Sending DeleteTablet for 3 replicas of tablet 5491b397f75647fa9f241703859c4070
I20260925 05:07:25.765626 13848 tablet_service.cc:1514] Processing DeleteTablet for tablet 5491b397f75647fa9f241703859c4070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:25 UTC) from {username='slave'} at 127.0.0.1:36772
I20260925 05:07:25.765828 13978 tablet_service.cc:1514] Processing DeleteTablet for tablet 5491b397f75647fa9f241703859c4070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:25 UTC) from {username='slave'} at 127.0.0.1:47806
I20260925 05:07:25.765877 14108 tablet_service.cc:1514] Processing DeleteTablet for tablet 5491b397f75647fa9f241703859c4070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:25 UTC) from {username='slave'} at 127.0.0.1:39338
I20260925 05:07:25.767663 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13786
I20260925 05:07:25.777137 14217 tablet_replica.cc:330] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: stopping tablet replica
I20260925 05:07:25.777602 14217 raft_consensus.cc:2229] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:25.778126 14217 raft_consensus.cc:2258] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:25.778564 14215 tablet_replica.cc:330] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: stopping tablet replica
I20260925 05:07:25.778965 14216 tablet_replica.cc:330] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: stopping tablet replica
I20260925 05:07:25.779207 14215 raft_consensus.cc:2229] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:25.779383 14216 raft_consensus.cc:2229] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:25.779874 14215 raft_consensus.cc:2258] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:25.779995 14216 raft_consensus.cc:2258] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:25.782373 14217 ts_tablet_manager.cc:1905] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:25.782385 14215 ts_tablet_manager.cc:1905] T 5491b397f75647fa9f241703859c4070 P 2168920747984e4288d718dabe44a4aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:25.783416 14187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.768966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.8.215.1:40717)
0925 05:07:25.769149 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.769158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.780903 (+ 11745us) server_negotiation.cc:407] Connection header received
0925 05:07:25.781203 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.781208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.781290 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.781367 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.782636 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.782641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.782647 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.782665 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.782699 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.782713 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.782807 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.783067 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.783129 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:07:25.784515 13908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:25.785764 14216 ts_tablet_manager.cc:1905] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:25.790441 14217 ts_tablet_manager.cc:1918] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:25.790633 14217 log.cc:1196] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/5491b397f75647fa9f241703859c4070
I20260925 05:07:25.791280 14217 ts_tablet_manager.cc:1939] T 5491b397f75647fa9f241703859c4070 P e34ba42676f7422dba2b87c23335767c: Deleting consensus metadata
I20260925 05:07:25.793154 14216 ts_tablet_manager.cc:1918] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:25.793390 14216 log.cc:1196] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/5491b397f75647fa9f241703859c4070
I20260925 05:07:25.793584 13714 catalog_manager.cc:4841] TS e34ba42676f7422dba2b87c23335767c (127.8.215.3:34091): tablet 5491b397f75647fa9f241703859c4070 (table test-table [id=cbc3675865dd4c4bb0ac5a17ffa84834]) successfully deleted
I20260925 05:07:25.794056 14216 ts_tablet_manager.cc:1939] T 5491b397f75647fa9f241703859c4070 P cf248985c8204b84a6ea4679c4334b12: Deleting consensus metadata
I20260925 05:07:25.795500 13714 catalog_manager.cc:4841] TS cf248985c8204b84a6ea4679c4334b12 (127.8.215.2:33185): tablet 5491b397f75647fa9f241703859c4070 (table test-table [id=cbc3675865dd4c4bb0ac5a17ffa84834]) successfully deleted
I20260925 05:07:25.942543 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13786
W20260925 05:07:25.960664 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.215.1:40717 (error 108)
I20260925 05:07:25.961000 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13916
I20260925 05:07:25.965991 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.962631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.8.215.2:33185)
0925 05:07:25.962904 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:25.962913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:25.962938 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:07:25.963053 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:25.963061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:25.963143 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:25.963259 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:25.964675 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:25.964685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:25.964693 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:25.964740 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:25.964783 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:25.964804 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:25.964954 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:25.965474 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 05:07:25.965549 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 05:07:25.967487 14038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:25.974129 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:25.973252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36784)
0925 05:07:25.973757 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:25.973886 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
W20260925 05:07:25.974500 13713 catalog_manager.cc:4584] TS 2168920747984e4288d718dabe44a4aa (127.8.215.1:40717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5491b397f75647fa9f241703859c4070: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
I20260925 05:07:26.028319 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.027750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36788)
0925 05:07:26.028070 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.028161 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 05:07:26.029431 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.029055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36804)
0925 05:07:26.029219 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.029304 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 05:07:26.082561 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13916
I20260925 05:07:26.089994 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.089518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36816)
0925 05:07:26.089764 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.089851 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:07:26.091216 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.090725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36828)
0925 05:07:26.090929 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.091027 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 05:07:26.102491 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14046
I20260925 05:07:26.105098 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.103209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.8.215.3:34091)
0925 05:07:26.103367 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:26.103371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:26.103391 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:07:26.103557 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:26.103561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:26.103647 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:26.103737 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:26.104593 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:26.104597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:26.104600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:26.104612 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:26.104635 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:26.104645 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:26.104793 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:26.104829 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 05:07:26.104870 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:07:26.105664 14168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:26.172037 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.171543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36840)
0925 05:07:26.171777 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.171879 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 05:07:26.173157 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.172769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:40717 (local address 127.0.0.1:36848)
0925 05:07:26.172950 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.173032 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 05:07:26.191500 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14046
I20260925 05:07:26.207829 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 13696
I20260925 05:07:26.209003 13757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:26.312197 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 13696
2026-09-25T05:07:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2822 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 05:07:26.353506 9052 mini_oidc.cc:152] Starting JWKS server
I20260925 05:07:26.355983 9052 webserver.cc:460] Webserver started at https://127.0.0.1:40753/ using document root <none> and password file <none>
I20260925 05:07:26.356243 9052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:07:26.356853 9052 webserver.cc:460] Webserver started at http://127.0.0.1:43645/ using document root <none> and password file <none>
2026-09-25T05:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:26Z Disabled control of system clock
I20260925 05:07:26.383074 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35679
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:35679
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40753/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/testchainca.cert with env {}
W20260925 05:07:26.699383 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:26.699661 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:26.699744 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:26.706665 14230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:26.706763 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:26.706799 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:26.706851 14230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:26.706894 14230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:07:26.706952 14230 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:07:26.706991 14230 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40753/jwks.json
I20260925 05:07:26.716524 14230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:35679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40753/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:26.717753 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:26.719013 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:26.730719 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:26.730826 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:26.730810 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:26.731401 14230 server_base.cc:1034] running on GCE node
I20260925 05:07:26.732128 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:26.733558 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:26.734787 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790312846734755 us; error 44 us; skew 500 ppm
I20260925 05:07:26.735133 14230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:26.737426 14230 webserver.cc:460] Webserver started at http://127.8.215.62:36569/ using document root <none> and password file <none>
I20260925 05:07:26.738090 14230 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:26.738222 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:26.738533 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:26.741353 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "d8d14d92d6fb4a608aeef4b00f63675d"
format_stamp: "Formatted at 2026-09-25 05:07:26 on dist-test-slave-wpdm"
server_key: "9d07d40c23ef58c480d6e56f2b90bad3"
server_key_iv: "669ab92e4113672b21ae9e083a6816a5"
server_key_version: "encryptionkey@0"
I20260925 05:07:26.742126 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "d8d14d92d6fb4a608aeef4b00f63675d"
format_stamp: "Formatted at 2026-09-25 05:07:26 on dist-test-slave-wpdm"
server_key: "9d07d40c23ef58c480d6e56f2b90bad3"
server_key_iv: "669ab92e4113672b21ae9e083a6816a5"
server_key_version: "encryptionkey@0"
I20260925 05:07:26.746989 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:07:26.750295 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:26.751631 14230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:07:26.751812 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "d8d14d92d6fb4a608aeef4b00f63675d"
format_stamp: "Formatted at 2026-09-25 05:07:26 on dist-test-slave-wpdm"
server_key: "9d07d40c23ef58c480d6e56f2b90bad3"
server_key_iv: "669ab92e4113672b21ae9e083a6816a5"
server_key_version: "encryptionkey@0"
I20260925 05:07:26.752028 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:26.782532 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:26.783378 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:26.783707 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:26.801659 14230 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:35679
I20260925 05:07:26.801683 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:35679 every 8 connection(s)
I20260925 05:07:26.802977 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:26.806497 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:26.810554 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14230
I20260925 05:07:26.810766 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:26.811367 9052 external_mini_cluster.cc:1428] Setting key b72dfe2609c572eeaafccf4501ba90f9
I20260925 05:07:26.817247 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d: Bootstrap starting.
I20260925 05:07:26.820659 14296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:26.821902 14296 log.cc:824] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:26.824972 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d: No bootstrap required, opened a new log
I20260925 05:07:26.825045 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.813124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.8.215.62:35679)
0925 05:07:26.814059 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:26.814072 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:26.814112 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:07:26.815788 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:26.815808 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:26.816214 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:26.816516 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:26.817200 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:26.818515 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:26.820455 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:26.821167 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:26.821302 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:26.822655 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:26.822664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:26.822687 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:26.823383 (+ 696us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:26.823463 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:26.823482 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:26.823568 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:26.823793 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 05:07:26.824103 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":323,"threads_started":1}
I20260925 05:07:26.828827 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:07:26.833063 14296 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35679 } }
I20260925 05:07:26.833417 14296 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:26.833566 14296 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d14d92d6fb4a608aeef4b00f63675d, State: Initialized, Role: FOLLOWER
I20260925 05:07:26.834105 14296 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35679 } }
I20260925 05:07:26.834342 14296 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:26.834487 14296 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:26.834654 14296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:26.836992 14296 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35679 } }
I20260925 05:07:26.837476 14296 leader_election.cc:302] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d14d92d6fb4a608aeef4b00f63675d; no voters:
I20260925 05:07:26.838194 14296 leader_election.cc:288] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:26.838301 14301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:26.839151 14301 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [term 1 LEADER]: Becoming Leader. State: Replica: d8d14d92d6fb4a608aeef4b00f63675d, State: Running, Role: LEADER
I20260925 05:07:26.839659 14301 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35679 } }
I20260925 05:07:26.840870 14296 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:26.842584 14303 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d14d92d6fb4a608aeef4b00f63675d. Latest consensus state: current_term: 1 leader_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35679 } } }
I20260925 05:07:26.842564 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d14d92d6fb4a608aeef4b00f63675d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35679 } } }
I20260925 05:07:26.843070 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:26.843070 14303 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:26.845112 14306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:26.849884 14306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:26.860198 14306 catalog_manager.cc:1347] Generated new cluster ID: c7283f8d55f34e63982643a33fc0780e
I20260925 05:07:26.860342 14306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:26.876356 14306 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:26.878052 14306 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:26.884686 14306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d14d92d6fb4a608aeef4b00f63675d: Generated new TSK 0
I20260925 05:07:26.885574 14306 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:07:26.926589 14326 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 05:07:26.926761 14326 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:07:26.927481 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:26.899805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.8.215.62:35679)
0925 05:07:26.900008 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:26.900016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:26.900423 (+ 407us) server_negotiation.cc:407] Connection header received
0925 05:07:26.900775 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:26.900781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:26.900863 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:26.901010 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:07:26.901562 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:26.902285 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:26.903708 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:26.904151 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:26.904295 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:26.904843 (+ 548us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:07:26.926967 (+ 22124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:07:26.927255 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45490: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 05:07:26.927626 14299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45490: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 05:07:26.929425 14325 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:26.899724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:35679 (local address 127.0.0.1:45490)
0925 05:07:26.900322 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:26.900337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:26.900522 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:26.901059 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:26.901063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:26.901079 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:07:26.901393 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:26.901401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:26.902430 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:26.902436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:26.903557 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:26.903567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:26.904335 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:26.904343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:26.904480 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:26.904534 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:07:26.927266 (+ 22732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:07:26.927643 (+ 377us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.215.62:35679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":332,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.215.62:35679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 05:07:26.930850 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14230
I20260925 05:07:26.931967 14291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:27.026820 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14230
2026-09-25T05:07:27Z chronyd exiting
I20260925 05:07:27.078179 9052 test_util.cc:175] -----------------------------------------------
I20260925 05:07:27.078372 9052 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312801912195-9052-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 05:07:27.082181 9052 mini_oidc.cc:152] Starting JWKS server
I20260925 05:07:27.084038 9052 webserver.cc:460] Webserver started at https://127.0.0.1:36855/ using document root <none> and password file <none>
I20260925 05:07:27.084298 9052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:07:27.085053 9052 webserver.cc:460] Webserver started at http://127.0.0.1:40553/ using document root <none> and password file <none>
2026-09-25T05:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:27Z Disabled control of system clock
I20260925 05:07:27.115387 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40119
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:40119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36855/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/testchainca.cert with env {}
W20260925 05:07:27.433529 14333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:27.433871 14333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:27.434060 14333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:27.441658 14333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:27.441807 14333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:27.441877 14333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:27.441959 14333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:27.442040 14333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:07:27.442127 14333 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:07:27.442186 14333 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36855/jwks.json
I20260925 05:07:27.451857 14333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:40119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36855/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:27.453200 14333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:27.454573 14333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:27.465871 14341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:27.465819 14339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:27.465909 14338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:27.466850 14333 server_base.cc:1034] running on GCE node
I20260925 05:07:27.467597 14333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:27.469045 14333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:27.470263 14333 hybrid_clock.cc:648] HybridClock initialized: now 1790312847470229 us; error 46 us; skew 500 ppm
I20260925 05:07:27.470607 14333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:27.472676 14333 webserver.cc:460] Webserver started at http://127.8.215.62:45525/ using document root <none> and password file <none>
I20260925 05:07:27.473351 14333 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:27.473484 14333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:27.473799 14333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:27.476508 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "fbdfff353b3b483f8aebb98ebcdde104"
format_stamp: "Formatted at 2026-09-25 05:07:27 on dist-test-slave-wpdm"
server_key: "1461bc7a1bdd401185792770949b0696"
server_key_iv: "f2c377721ca38e05e2c53ba423719858"
server_key_version: "encryptionkey@0"
I20260925 05:07:27.477541 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "fbdfff353b3b483f8aebb98ebcdde104"
format_stamp: "Formatted at 2026-09-25 05:07:27 on dist-test-slave-wpdm"
server_key: "1461bc7a1bdd401185792770949b0696"
server_key_iv: "f2c377721ca38e05e2c53ba423719858"
server_key_version: "encryptionkey@0"
I20260925 05:07:27.482436 14333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:07:27.485633 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:27.486898 14333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:07:27.487087 14333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "fbdfff353b3b483f8aebb98ebcdde104"
format_stamp: "Formatted at 2026-09-25 05:07:27 on dist-test-slave-wpdm"
server_key: "1461bc7a1bdd401185792770949b0696"
server_key_iv: "f2c377721ca38e05e2c53ba423719858"
server_key_version: "encryptionkey@0"
I20260925 05:07:27.487295 14333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:27.513845 14333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:27.514739 14333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:27.515065 14333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:27.533499 14333 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:40119
I20260925 05:07:27.533540 14398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:40119 every 8 connection(s)
I20260925 05:07:27.534901 14333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:27.535427 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14333
I20260925 05:07:27.535624 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:27.536528 9052 external_mini_cluster.cc:1428] Setting key 3e4b965031f76a3baf530d5abeb12cbc
I20260925 05:07:27.539448 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:27.549731 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:27.538439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.8.215.62:40119)
0925 05:07:27.539401 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:27.539412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:27.539455 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:07:27.541134 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:27.541157 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:27.541622 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:27.542047 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:27.542608 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:27.543956 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:27.545336 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:27.545997 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:27.546196 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:27.547384 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:27.547392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:27.547405 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:27.547994 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:27.548068 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:27.548093 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:27.548215 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:27.548450 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 05:07:27.548824 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":340,"threads_started":1}
I20260925 05:07:27.549955 14399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104: Bootstrap starting.
I20260925 05:07:27.553777 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:07:27.553855 14399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:27.555049 14399 log.cc:824] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:27.558084 14399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104: No bootstrap required, opened a new log
I20260925 05:07:27.565186 14399 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdfff353b3b483f8aebb98ebcdde104" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40119 } }
I20260925 05:07:27.565527 14399 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:27.565598 14399 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdfff353b3b483f8aebb98ebcdde104, State: Initialized, Role: FOLLOWER
I20260925 05:07:27.566145 14399 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdfff353b3b483f8aebb98ebcdde104" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40119 } }
I20260925 05:07:27.566337 14399 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:27.566431 14399 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:27.566558 14399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:27.568617 14399 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdfff353b3b483f8aebb98ebcdde104" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40119 } }
I20260925 05:07:27.569151 14399 leader_election.cc:302] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbdfff353b3b483f8aebb98ebcdde104; no voters:
I20260925 05:07:27.569648 14399 leader_election.cc:288] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:27.569818 14404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:27.570986 14404 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [term 1 LEADER]: Becoming Leader. State: Replica: fbdfff353b3b483f8aebb98ebcdde104, State: Running, Role: LEADER
I20260925 05:07:27.571607 14404 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdfff353b3b483f8aebb98ebcdde104" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40119 } }
I20260925 05:07:27.571930 14399 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:27.574704 14405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbdfff353b3b483f8aebb98ebcdde104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdfff353b3b483f8aebb98ebcdde104" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40119 } } }
I20260925 05:07:27.574692 14406 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdfff353b3b483f8aebb98ebcdde104. Latest consensus state: current_term: 1 leader_uuid: "fbdfff353b3b483f8aebb98ebcdde104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdfff353b3b483f8aebb98ebcdde104" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40119 } } }
I20260925 05:07:27.575064 14406 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:27.575063 14405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:27.576097 14414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:27.580924 14414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:27.589329 14414 catalog_manager.cc:1347] Generated new cluster ID: 36482f961064406eb99dbcdb377bd0d1
I20260925 05:07:27.589438 14414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:27.600227 14414 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:27.601768 14414 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:27.610153 14414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbdfff353b3b483f8aebb98ebcdde104: Generated new TSK 0
I20260925 05:07:27.610980 14414 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:07:27.641260 14429 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 05:07:27.641409 14429 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:07:27.642670 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:27.622708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.8.215.62:40119)
0925 05:07:27.622977 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:27.622987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:27.623488 (+ 501us) server_negotiation.cc:407] Connection header received
0925 05:07:27.623682 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:27.623688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:27.623793 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:27.623959 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:07:27.624361 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:27.625524 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:27.626859 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:27.627518 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:27.627653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:27.628147 (+ 494us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:07:27.641937 (+ 13790us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:07:27.642347 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 05:07:27.642688 14428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:27.622508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40119 (local address 127.0.0.1:38842)
0925 05:07:27.623371 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:27.623387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:27.623511 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:27.623975 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:27.623978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:27.623987 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:07:27.624217 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:27.624224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:27.625660 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:27.625664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:27.626719 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:27.626731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:27.627683 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:27.627687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:27.627824 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:27.627857 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:07:27.642190 (+ 14333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:07:27.642381 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.215.62:40119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":616,"threads_started":1}
W20260925 05:07:27.642800 14402 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 05:07:27.643778 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14333
I20260925 05:07:27.644984 14394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:27.734674 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14333
2026-09-25T05:07:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (707 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 05:07:27.789188 9052 mini_oidc.cc:152] Starting JWKS server
I20260925 05:07:27.791157 9052 webserver.cc:460] Webserver started at https://127.0.0.1:38795/ using document root <none> and password file <none>
I20260925 05:07:27.791414 9052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:07:27.792021 9052 webserver.cc:460] Webserver started at http://127.0.0.1:35573/ using document root <none> and password file <none>
2026-09-25T05:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:27Z Disabled control of system clock
I20260925 05:07:27.819152 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44479
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:44479
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38795/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:07:28.145576 14436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:28.145931 14436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:28.146083 14436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:28.153221 14436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:28.153378 14436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:28.153450 14436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:28.153535 14436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:28.153609 14436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:07:28.153733 14436 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:07:28.153796 14436 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38795/jwks.json
I20260925 05:07:28.162955 14436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:44479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38795/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:28.164191 14436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:28.165874 14436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:28.177649 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:28.177541 14444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:28.177536 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:28.178321 14436 server_base.cc:1034] running on GCE node
I20260925 05:07:28.179075 14436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:28.180400 14436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:28.181622 14436 hybrid_clock.cc:648] HybridClock initialized: now 1790312848181568 us; error 47 us; skew 500 ppm
I20260925 05:07:28.181957 14436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:28.184212 14436 webserver.cc:460] Webserver started at http://127.8.215.62:33749/ using document root <none> and password file <none>
I20260925 05:07:28.184964 14436 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:28.185103 14436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:28.185434 14436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:28.188241 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "579bde1fe67348bfac35181eb8726189"
format_stamp: "Formatted at 2026-09-25 05:07:28 on dist-test-slave-wpdm"
server_key: "eedee7f3d577950cefe2fae5d47c6a60"
server_key_iv: "9834a1b72387ebc406e7619a137e0f3b"
server_key_version: "encryptionkey@0"
I20260925 05:07:28.189091 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "579bde1fe67348bfac35181eb8726189"
format_stamp: "Formatted at 2026-09-25 05:07:28 on dist-test-slave-wpdm"
server_key: "eedee7f3d577950cefe2fae5d47c6a60"
server_key_iv: "9834a1b72387ebc406e7619a137e0f3b"
server_key_version: "encryptionkey@0"
I20260925 05:07:28.194259 14436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:07:28.197551 14450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:28.198925 14436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:28.199110 14436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "579bde1fe67348bfac35181eb8726189"
format_stamp: "Formatted at 2026-09-25 05:07:28 on dist-test-slave-wpdm"
server_key: "eedee7f3d577950cefe2fae5d47c6a60"
server_key_iv: "9834a1b72387ebc406e7619a137e0f3b"
server_key_version: "encryptionkey@0"
I20260925 05:07:28.199321 14436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:28.222126 14436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:28.222982 14436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:28.223310 14436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:28.241971 14436 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:44479
I20260925 05:07:28.241990 14501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:44479 every 8 connection(s)
I20260925 05:07:28.243405 14436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:28.243968 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14436
I20260925 05:07:28.244194 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:28.245044 9052 external_mini_cluster.cc:1428] Setting key c4f4cdd9ff5dbf26c5c8d0cffe56404a
I20260925 05:07:28.247892 14502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:28.259083 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:28.246972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.8.215.62:44479)
0925 05:07:28.247985 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:28.248002 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:28.248060 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:07:28.249702 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:28.249723 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:28.250173 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:28.250603 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:28.251157 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:28.252459 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:28.253909 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:28.254939 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:28.255148 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:28.256228 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:28.256239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:28.256257 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:28.257040 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:28.257139 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:28.257166 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:28.257292 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:28.257490 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 05:07:28.257839 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":357,"threads_started":1}
I20260925 05:07:28.260218 14502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189: Bootstrap starting.
I20260925 05:07:28.262964 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:07:28.263573 14502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:28.264531 14502 log.cc:824] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:28.267714 14502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189: No bootstrap required, opened a new log
I20260925 05:07:28.275683 14502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579bde1fe67348bfac35181eb8726189" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44479 } }
I20260925 05:07:28.276054 14502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:28.276136 14502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 579bde1fe67348bfac35181eb8726189, State: Initialized, Role: FOLLOWER
I20260925 05:07:28.276645 14502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579bde1fe67348bfac35181eb8726189" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44479 } }
I20260925 05:07:28.276947 14502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:28.277058 14502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:28.277189 14502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:28.279449 14502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579bde1fe67348bfac35181eb8726189" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44479 } }
I20260925 05:07:28.280033 14502 leader_election.cc:302] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 579bde1fe67348bfac35181eb8726189; no voters:
I20260925 05:07:28.280617 14502 leader_election.cc:288] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:28.280833 14507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:28.281623 14507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [term 1 LEADER]: Becoming Leader. State: Replica: 579bde1fe67348bfac35181eb8726189, State: Running, Role: LEADER
I20260925 05:07:28.282174 14507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579bde1fe67348bfac35181eb8726189" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44479 } }
I20260925 05:07:28.282766 14502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:28.285213 14508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "579bde1fe67348bfac35181eb8726189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579bde1fe67348bfac35181eb8726189" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44479 } } }
I20260925 05:07:28.285567 14508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:28.285306 14509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 579bde1fe67348bfac35181eb8726189. Latest consensus state: current_term: 1 leader_uuid: "579bde1fe67348bfac35181eb8726189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579bde1fe67348bfac35181eb8726189" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44479 } } }
I20260925 05:07:28.285885 14509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:28.291342 14514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:28.295804 14514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:28.304430 14514 catalog_manager.cc:1347] Generated new cluster ID: 4452fa0cba8644b3874945860cc67c31
I20260925 05:07:28.304545 14514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:28.335709 14514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:28.337313 14514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:28.351751 14514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 579bde1fe67348bfac35181eb8726189: Generated new TSK 0
I20260925 05:07:28.352679 14514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:07:28.377031 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:28.367670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.8.215.62:44479)
0925 05:07:28.367912 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:28.367919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:28.368412 (+ 493us) server_negotiation.cc:407] Connection header received
0925 05:07:28.368638 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:28.368643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:28.368716 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:28.368920 (+ 204us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:07:28.369504 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:28.370242 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:28.372527 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:28.373171 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:28.373447 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:28.373996 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:07:28.376411 (+ 2415us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:07:28.376713 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 05:07:28.377154 14505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 05:07:28.379302 14531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:28.367471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44479 (local address 127.0.0.1:52298)
0925 05:07:28.368219 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:28.368242 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:28.368412 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:28.368967 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:28.368973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:28.368991 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:07:28.369324 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:28.369337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:28.370386 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:28.370393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:28.372345 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:28.372364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:28.373384 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:28.373392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:28.373592 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:28.373644 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:07:28.376629 (+ 2985us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:07:28.378797 (+ 2168us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.215.62:44479: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":1798,"thread_start_us":376,"threads_started":1}
I20260925 05:07:28.381297 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14436
I20260925 05:07:28.382494 14497 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 05:07:28.389207 14532 webserver.cc:533] Webserver: sslize failed:
E20260925 05:07:28.389288 14532 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:07:28.474694 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14436
2026-09-25T05:07:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T05:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:28Z Disabled control of system clock
I20260925 05:07:28.557531 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:37677
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:37677
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/testchainca.cert with env {}
W20260925 05:07:28.899302 14537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:28.899601 14537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:28.899773 14537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:28.907163 14537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:28.907274 14537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:28.907366 14537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:28.907467 14537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:28.907552 14537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:07:28.907663 14537 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:07:28.907738 14537 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 05:07:28.918140 14537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:37677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:28.919533 14537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:28.920949 14537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:28.933035 14543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:28.933090 14545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:28.933065 14542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:28.933925 14537 server_base.cc:1034] running on GCE node
I20260925 05:07:28.935016 14537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:28.937395 14537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:28.938697 14537 hybrid_clock.cc:648] HybridClock initialized: now 1790312848938612 us; error 55 us; skew 500 ppm
I20260925 05:07:28.939173 14537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:07:28.942514 14537 webserver.cc:460] Webserver started at http://127.8.215.62:34087/ using document root <none> and password file <none>
I20260925 05:07:28.943449 14537 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:28.943568 14537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:28.943893 14537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:28.947080 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "c8f23857bfec4b2db96cfc26b779652a"
format_stamp: "Formatted at 2026-09-25 05:07:28 on dist-test-slave-wpdm"
server_key: "b926aaed907de35827baa30e96353adc"
server_key_iv: "935952f043a2f56c3ad2389f0c5eee70"
server_key_version: "encryptionkey@0"
I20260925 05:07:28.947856 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "c8f23857bfec4b2db96cfc26b779652a"
format_stamp: "Formatted at 2026-09-25 05:07:28 on dist-test-slave-wpdm"
server_key: "b926aaed907de35827baa30e96353adc"
server_key_iv: "935952f043a2f56c3ad2389f0c5eee70"
server_key_version: "encryptionkey@0"
I20260925 05:07:28.952994 14537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:07:28.956465 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:28.957980 14537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:28.958166 14537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "c8f23857bfec4b2db96cfc26b779652a"
format_stamp: "Formatted at 2026-09-25 05:07:28 on dist-test-slave-wpdm"
server_key: "b926aaed907de35827baa30e96353adc"
server_key_iv: "935952f043a2f56c3ad2389f0c5eee70"
server_key_version: "encryptionkey@0"
I20260925 05:07:28.958376 14537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:28.969973 14537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:28.970744 14537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:28.971025 14537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:28.989262 14537 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:37677
I20260925 05:07:28.989287 14602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:37677 every 8 connection(s)
I20260925 05:07:28.990618 14537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:28.994244 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:28.997387 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14537
I20260925 05:07:28.997593 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:28.998193 9052 external_mini_cluster.cc:1428] Setting key 930c80c7ba57c9720d908924bc1f10f6
I20260925 05:07:29.005808 14603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a: Bootstrap starting.
I20260925 05:07:29.008802 14603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:29.009833 14603 log.cc:824] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:29.011199 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:29.000004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.8.215.62:37677)
0925 05:07:29.000967 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:29.000978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:29.001030 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:07:29.002529 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:29.002550 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:29.003019 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:29.003422 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:29.004035 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:29.005372 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:29.006713 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:29.007377 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:29.007591 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:29.008786 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:29.008796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:29.008813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:07:29.009399 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:29.009487 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:07:29.009506 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:29.009630 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:29.009852 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 05:07:29.010178 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":285,"threads_started":1}
I20260925 05:07:29.012678 14603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a: No bootstrap required, opened a new log
I20260925 05:07:29.015043 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:07:29.021677 14603 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f23857bfec4b2db96cfc26b779652a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 37677 } }
I20260925 05:07:29.022110 14603 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:29.022221 14603 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f23857bfec4b2db96cfc26b779652a, State: Initialized, Role: FOLLOWER
I20260925 05:07:29.022838 14603 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f23857bfec4b2db96cfc26b779652a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 37677 } }
I20260925 05:07:29.023036 14603 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:29.023133 14603 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:29.023252 14603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:29.025369 14603 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f23857bfec4b2db96cfc26b779652a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 37677 } }
I20260925 05:07:29.025884 14603 leader_election.cc:302] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f23857bfec4b2db96cfc26b779652a; no voters:
I20260925 05:07:29.026539 14603 leader_election.cc:288] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:29.026655 14608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:29.027930 14608 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [term 1 LEADER]: Becoming Leader. State: Replica: c8f23857bfec4b2db96cfc26b779652a, State: Running, Role: LEADER
I20260925 05:07:29.028486 14608 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f23857bfec4b2db96cfc26b779652a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 37677 } }
I20260925 05:07:29.029114 14603 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:29.031893 14609 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f23857bfec4b2db96cfc26b779652a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f23857bfec4b2db96cfc26b779652a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 37677 } } }
I20260925 05:07:29.032591 14609 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:29.032320 14610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f23857bfec4b2db96cfc26b779652a. Latest consensus state: current_term: 1 leader_uuid: "c8f23857bfec4b2db96cfc26b779652a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f23857bfec4b2db96cfc26b779652a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 37677 } } }
I20260925 05:07:29.033236 14610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:29.037708 14616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:29.043329 14616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:29.052784 14616 catalog_manager.cc:1347] Generated new cluster ID: 6cbc1a9c574241d9b04e55b2b5e88290
I20260925 05:07:29.052942 14616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:29.060881 14616 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:29.062203 14616 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:29.078168 14616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f23857bfec4b2db96cfc26b779652a: Generated new TSK 0
I20260925 05:07:29.079248 14616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:07:29.578013 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:29.099591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.8.215.62:37677)
0925 05:07:29.099790 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:29.099797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:29.100077 (+ 280us) server_negotiation.cc:407] Connection header received
0925 05:07:29.100347 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:29.100351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:29.100421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:29.100583 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:07:29.101113 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:29.101859 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:29.103250 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:29.103737 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:29.103883 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:29.104364 (+ 481us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:07:29.577214 (+472850us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:07:29.577681 (+ 467us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44722: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 05:07:29.578148 14606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44722: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 05:07:29.578223 14632 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:29.099422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:37677 (local address 127.0.0.1:44722)
0925 05:07:29.099963 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:29.099983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:29.100154 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:29.100593 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:29.100598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:29.100610 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:07:29.100961 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:29.100970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:29.102023 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:29.102029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:29.103103 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:29.103114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:29.103918 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:29.103924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:29.104073 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:29.104106 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:07:29.577515 (+473409us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:07:29.577824 (+ 309us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.215.62:37677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":382,"threads_started":1}
I20260925 05:07:29.579411 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14537
I20260925 05:07:29.580585 14598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:29.679970 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14537
2026-09-25T05:07:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1186 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (405 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:30 dist-test-slave-wpdm krb5kdc[14647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:30 dist-test-slave-wpdm krb5kdc[14647](info): set up 2 sockets
Sep 25 05:07:30 dist-test-slave-wpdm krb5kdc[14647](info): commencing operation
krb5kdc: starting...
W20260925 05:07:31.577888 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:31 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:31.785197 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:39277
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:39277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:32.102317 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:32.102635 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:32.102820 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:32.109897 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:32.109997 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:32.110122 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:32.110217 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:32.110283 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:32.119910 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:39277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:39277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:32.121209 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:32.122467 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:32.134346 14665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:32.134344 14666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:32.134622 14668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:32.135478 14660 server_base.cc:1034] running on GCE node
I20260925 05:07:32.136224 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:32.137662 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:32.138859 14660 hybrid_clock.cc:648] HybridClock initialized: now 1790312852138805 us; error 44 us; skew 500 ppm
I20260925 05:07:32.139196 14660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:32 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312852, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:32.142397 14660 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:32.143821 14660 webserver.cc:460] Webserver started at http://127.8.215.62:45205/ using document root <none> and password file <none>
I20260925 05:07:32.144441 14660 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:32.144538 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:32.144865 14660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:32.147701 14660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "40618c5a3259448db059cd9d98ee530a"
format_stamp: "Formatted at 2026-09-25 05:07:32 on dist-test-slave-wpdm"
server_key: "3188ccb30a7af9d90fcf576adb50d353"
server_key_iv: "7f0cf77051bc2d21111e2b36baa6e2f6"
server_key_version: "encryptionkey@0"
I20260925 05:07:32.148397 14660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "40618c5a3259448db059cd9d98ee530a"
format_stamp: "Formatted at 2026-09-25 05:07:32 on dist-test-slave-wpdm"
server_key: "3188ccb30a7af9d90fcf576adb50d353"
server_key_iv: "7f0cf77051bc2d21111e2b36baa6e2f6"
server_key_version: "encryptionkey@0"
I20260925 05:07:32.153362 14660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 05:07:32.156497 14675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:32.157799 14660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:32.157977 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "40618c5a3259448db059cd9d98ee530a"
format_stamp: "Formatted at 2026-09-25 05:07:32 on dist-test-slave-wpdm"
server_key: "3188ccb30a7af9d90fcf576adb50d353"
server_key_iv: "7f0cf77051bc2d21111e2b36baa6e2f6"
server_key_version: "encryptionkey@0"
I20260925 05:07:32.158188 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:32.171568 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:32.175387 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:32.175683 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:32.193221 14660 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:39277
I20260925 05:07:32.193243 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:39277 every 8 connection(s)
I20260925 05:07:32.194587 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:32.198125 14727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:32.204375 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14660
I20260925 05:07:32.204591 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:32.205436 9052 external_mini_cluster.cc:1428] Setting key 1ba2e6992050d3f325e57d40f17af979
I20260925 05:07:32.209124 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a: Bootstrap starting.
I20260925 05:07:32.212559 14727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:32.213749 14727 log.cc:824] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:32.216614 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a: No bootstrap required, opened a new log
Sep 25 05:07:32 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:32.225600 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40618c5a3259448db059cd9d98ee530a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39277 } }
I20260925 05:07:32.225991 14727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:32.226119 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40618c5a3259448db059cd9d98ee530a, State: Initialized, Role: FOLLOWER
I20260925 05:07:32.226747 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40618c5a3259448db059cd9d98ee530a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39277 } }
I20260925 05:07:32.226998 14727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:32.227147 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:32.227335 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:32.229336 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:32.207607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.8.215.62:39277)
0925 05:07:32.208352 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:32.208363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:32.208402 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:07:32.210187 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:32.210210 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:32.210601 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:32.210971 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:32.212765 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:32.214250 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:32.215862 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:32.216572 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:32.216793 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:32.221368 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:32.221375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:32.221392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:32.221425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:32.224562 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:32.225329 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:32.225337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:32.225343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:32.225495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:32.225877 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:32.225883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:32.225886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:32.226561 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:32.226893 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:32.227268 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 05:07:32.227900 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":361,"threads_started":1}
I20260925 05:07:32.230363 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40618c5a3259448db059cd9d98ee530a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39277 } }
I20260925 05:07:32.230877 14727 leader_election.cc:302] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40618c5a3259448db059cd9d98ee530a; no voters:
I20260925 05:07:32.231642 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:32.231828 14732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:32.232919 14732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [term 1 LEADER]: Becoming Leader. State: Replica: 40618c5a3259448db059cd9d98ee530a, State: Running, Role: LEADER
I20260925 05:07:32.233446 14732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40618c5a3259448db059cd9d98ee530a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39277 } }
I20260925 05:07:32.234112 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:32.236598 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40618c5a3259448db059cd9d98ee530a. Latest consensus state: current_term: 1 leader_uuid: "40618c5a3259448db059cd9d98ee530a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40618c5a3259448db059cd9d98ee530a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39277 } } }
I20260925 05:07:32.237118 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:32.236555 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40618c5a3259448db059cd9d98ee530a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40618c5a3259448db059cd9d98ee530a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39277 } } }
I20260925 05:07:32.237684 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:32.239261 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:32.244704 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:32.252395 14738 catalog_manager.cc:1347] Generated new cluster ID: de7b17d4acbc4d2d9678e374097c8b43
I20260925 05:07:32.252514 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:32.265374 14738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:32.267215 14738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:32.278191 14738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40618c5a3259448db059cd9d98ee530a: Generated new TSK 0
I20260925 05:07:32.279179 14738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:07:32.329582 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:39277
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:32.652757 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:32.653033 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:32.653249 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:32.660454 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:32.660585 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:32.660775 14753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:32.671029 14753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:39277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:32.672312 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:32.673615 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:32.686947 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:32.687031 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:32.687005 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:32.687724 14753 server_base.cc:1034] running on GCE node
I20260925 05:07:32.688328 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:32.689313 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:32.690608 14753 hybrid_clock.cc:648] HybridClock initialized: now 1790312852690546 us; error 68 us; skew 500 ppm
I20260925 05:07:32.690984 14753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:32 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312852, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:32.694101 14753 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:07:32.695546 14753 webserver.cc:460] Webserver started at http://127.8.215.1:37283/ using document root <none> and password file <none>
I20260925 05:07:32.696168 14753 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:32.696261 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:32.696514 14753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:32.699419 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "7c00940f4b2a45c98d62e92da56856c0"
format_stamp: "Formatted at 2026-09-25 05:07:32 on dist-test-slave-wpdm"
server_key: "472926eac79fc868097fb3ad97ea51a7"
server_key_iv: "edf5d550aa556987ef72c34a796a7fab"
server_key_version: "encryptionkey@0"
I20260925 05:07:32.700117 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "7c00940f4b2a45c98d62e92da56856c0"
format_stamp: "Formatted at 2026-09-25 05:07:32 on dist-test-slave-wpdm"
server_key: "472926eac79fc868097fb3ad97ea51a7"
server_key_iv: "edf5d550aa556987ef72c34a796a7fab"
server_key_version: "encryptionkey@0"
I20260925 05:07:32.704988 14753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 05:07:32.708259 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:32.709667 14753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:32.709820 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "7c00940f4b2a45c98d62e92da56856c0"
format_stamp: "Formatted at 2026-09-25 05:07:32 on dist-test-slave-wpdm"
server_key: "472926eac79fc868097fb3ad97ea51a7"
server_key_iv: "edf5d550aa556987ef72c34a796a7fab"
server_key_version: "encryptionkey@0"
I20260925 05:07:32.710044 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:32.747779 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:32.752460 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:32.752784 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:32.753780 14753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:32.755143 14753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:32.755214 14753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:32.755394 14753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:32.755458 14753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:32.788704 14753 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:34491
I20260925 05:07:32.788857 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:34491 every 8 connection(s)
I20260925 05:07:32.790086 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:32.800299 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14753
I20260925 05:07:32.800523 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:32.801388 9052 external_mini_cluster.cc:1428] Setting key 6d030cc0edb5e24223559987bdc07b8d
Sep 25 05:07:32 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312852, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:32.811496 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:32.792701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:50517 (local address 127.8.215.62:39277)
0925 05:07:32.793238 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:32.793245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:32.794259 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 05:07:32.795989 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:32.795994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:32.796062 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:32.796164 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:32.797764 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:32.798542 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:32.800030 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:32.800765 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:32.800920 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:32.804458 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:32.804468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:32.804476 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:32.804524 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:32.808330 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:32.809096 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:32.809105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:32.809112 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:32.809267 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:32.809673 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:32.809682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:32.809688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:32.810101 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:32.810252 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:32.810957 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 05:07:32.811225 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":224,"threads_started":1}
I20260925 05:07:32.812709 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:32.793374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:39277 (local address 127.8.215.1:50517)
0925 05:07:32.794096 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:32.794137 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:32.795740 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:32.796442 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:32.796450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:32.797029 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:32.797597 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:32.797614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:32.798683 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:32.798687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:32.799877 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:32.799887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:32.800971 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:32.800977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:32.801171 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:32.801847 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:32.801873 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:32.804238 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:32.808482 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:32.808488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:32.808500 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:32.808982 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:32.809374 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:32.809377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:32.809380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:32.809543 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:32.810225 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:32.810231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:32.810550 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 05:07:32.810774 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 05:07:32.811214 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":293,"threads_started":1}
I20260925 05:07:32.814389 14881 heartbeater.cc:347] Connected to a master server at 127.8.215.62:39277
I20260925 05:07:32.814757 14881 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:32.815511 14881 heartbeater.cc:511] Master 127.8.215.62:39277 requested a full tablet report, sending...
I20260925 05:07:32.817664 14692 ts_manager.cc:194] Registered new tserver with Master: 7c00940f4b2a45c98d62e92da56856c0 (127.8.215.1:34491)
I20260925 05:07:32.819162 14692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:50517
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:07:32.852674 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:39277
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:33.178489 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:33.178740 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:33.178893 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:33.185972 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:33.186081 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:33.186185 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:33.195559 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:39277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:33.196707 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:33.197973 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:33.210773 14892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:33.210850 14893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:33.211316 14895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:33.211673 14887 server_base.cc:1034] running on GCE node
I20260925 05:07:33.212255 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:33.213213 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:33.214495 14887 hybrid_clock.cc:648] HybridClock initialized: now 1790312853214427 us; error 42 us; skew 500 ppm
I20260925 05:07:33.214808 14887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.217916 14887 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:07:33.219314 14887 webserver.cc:460] Webserver started at http://127.8.215.2:46727/ using document root <none> and password file <none>
I20260925 05:07:33.219987 14887 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:33.220144 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:33.220466 14887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:33.223407 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "0ae21e1ca21d47e781ce5df1fe7b8c0d"
format_stamp: "Formatted at 2026-09-25 05:07:33 on dist-test-slave-wpdm"
server_key: "c4dd0708b3e6e377eff1428fc921b592"
server_key_iv: "195239f84b58376fcd47ba94d42e288b"
server_key_version: "encryptionkey@0"
I20260925 05:07:33.224167 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "0ae21e1ca21d47e781ce5df1fe7b8c0d"
format_stamp: "Formatted at 2026-09-25 05:07:33 on dist-test-slave-wpdm"
server_key: "c4dd0708b3e6e377eff1428fc921b592"
server_key_iv: "195239f84b58376fcd47ba94d42e288b"
server_key_version: "encryptionkey@0"
I20260925 05:07:33.229104 14887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 05:07:33.232455 14902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:33.233783 14887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:33.233971 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "0ae21e1ca21d47e781ce5df1fe7b8c0d"
format_stamp: "Formatted at 2026-09-25 05:07:33 on dist-test-slave-wpdm"
server_key: "c4dd0708b3e6e377eff1428fc921b592"
server_key_iv: "195239f84b58376fcd47ba94d42e288b"
server_key_version: "encryptionkey@0"
I20260925 05:07:33.234206 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:33.252573 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:33.256606 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:33.256948 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:33.257931 14887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:33.259276 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:33.259395 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:33.259570 14887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:33.259647 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:33.293994 14887 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:39961
I20260925 05:07:33.294032 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:39961 every 8 connection(s)
I20260925 05:07:33.295395 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:33.302500 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 14887
I20260925 05:07:33.302696 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:33.303259 9052 external_mini_cluster.cc:1428] Setting key eef72d2299ccc95dc5db68a5e30b9fb8
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:33.316246 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.297968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:57359 (local address 127.8.215.62:39277)
0925 05:07:33.298148 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.298152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.299272 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 05:07:33.301091 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.301097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.301186 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.301301 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.302769 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.303835 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.305599 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.306336 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.306485 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.309973 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:33.309982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:33.309990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:33.310034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:33.312961 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.313889 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.313897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.313903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.314037 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.314521 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.314524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.314526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.314877 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:33.314981 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:33.315835 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 05:07:33.315993 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:07:33.317824 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.298336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:39277 (local address 127.8.215.2:57359)
0925 05:07:33.299102 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:33.299140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:33.300821 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:33.301533 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:33.301542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:33.302040 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:33.302569 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:33.302585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.304005 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.304010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:33.305449 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:33.305460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.306535 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.306539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:33.306729 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.307390 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:33.307416 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:33.309750 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:33.313085 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:33.313097 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:33.313116 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:33.313730 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:33.314169 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:33.314176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:33.314183 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:33.314415 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:33.315002 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:33.315013 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:33.315446 (+ 433us) client_negotiation.cc:769] Sending connection context
0925 05:07:33.315760 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 05:07:33.316326 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":319,"threads_started":1}
I20260925 05:07:33.319327 15015 heartbeater.cc:347] Connected to a master server at 127.8.215.62:39277
I20260925 05:07:33.319617 15015 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:33.320340 15015 heartbeater.cc:511] Master 127.8.215.62:39277 requested a full tablet report, sending...
I20260925 05:07:33.321954 14692 ts_manager.cc:194] Registered new tserver with Master: 0ae21e1ca21d47e781ce5df1fe7b8c0d (127.8.215.2:39961)
I20260925 05:07:33.323189 14692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:57359
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:07:33.351305 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:39277
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:33.668294 15020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:33.668553 15020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:33.668753 15020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:33.675793 15020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:33.675894 15020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:33.676085 15020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:33.685655 15020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:39277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:33.686878 15020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:33.688337 15020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:33.701458 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:33.701524 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:33.701547 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:33.702620 15020 server_base.cc:1034] running on GCE node
I20260925 05:07:33.703258 15020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:33.704126 15020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:33.705350 15020 hybrid_clock.cc:648] HybridClock initialized: now 1790312853705322 us; error 44 us; skew 500 ppm
I20260925 05:07:33.705684 15020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.708820 15020 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:07:33.710274 15020 webserver.cc:460] Webserver started at http://127.8.215.3:42413/ using document root <none> and password file <none>
I20260925 05:07:33.710945 15020 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:33.711092 15020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:33.711392 15020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:33.714388 15020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "517c43724ea44afaa47c74c79cc1d80b"
format_stamp: "Formatted at 2026-09-25 05:07:33 on dist-test-slave-wpdm"
server_key: "ad5a1171e0f856bf0e5fa337474a2cf6"
server_key_iv: "8808c28652b9635307b8fa06aeee52c8"
server_key_version: "encryptionkey@0"
I20260925 05:07:33.715140 15020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "517c43724ea44afaa47c74c79cc1d80b"
format_stamp: "Formatted at 2026-09-25 05:07:33 on dist-test-slave-wpdm"
server_key: "ad5a1171e0f856bf0e5fa337474a2cf6"
server_key_iv: "8808c28652b9635307b8fa06aeee52c8"
server_key_version: "encryptionkey@0"
I20260925 05:07:33.719995 15020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:07:33.723323 15035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:33.724534 15020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:33.724694 15020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "517c43724ea44afaa47c74c79cc1d80b"
format_stamp: "Formatted at 2026-09-25 05:07:33 on dist-test-slave-wpdm"
server_key: "ad5a1171e0f856bf0e5fa337474a2cf6"
server_key_iv: "8808c28652b9635307b8fa06aeee52c8"
server_key_version: "encryptionkey@0"
I20260925 05:07:33.724951 15020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:33.748975 15020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:33.753270 15020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:33.753552 15020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:33.754480 15020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:33.755716 15020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:33.755820 15020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:33.755968 15020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:33.756040 15020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:33.788540 15020 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:35569
I20260925 05:07:33.788596 15147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:35569 every 8 connection(s)
I20260925 05:07:33.789819 15020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:33.790148 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15020
I20260925 05:07:33.790308 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:33.790822 9052 external_mini_cluster.cc:1428] Setting key 87703b5bcad27c952475891d6d6006dc
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:33.810268 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.792856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:33519 (local address 127.8.215.62:39277)
0925 05:07:33.793093 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.793102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.794640 (+ 1538us) server_negotiation.cc:407] Connection header received
0925 05:07:33.796796 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.796800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.796885 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.796967 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.798511 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.799234 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.800760 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.801152 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.801245 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.804528 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:33.804533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:33.804536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:33.804567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:33.807417 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.808222 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.808226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.808228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.808343 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.808798 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.808801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.808803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.809047 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:33.809164 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:33.809845 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 05:07:33.809990 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:07:33.811287 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.793432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:39277 (local address 127.8.215.3:33519)
0925 05:07:33.794457 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:33.794526 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:33.796515 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:33.797237 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:33.797245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:33.797794 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:33.798361 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:33.798377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.799348 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.799352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:33.800606 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:33.800618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.801370 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.801374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:33.801596 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.802200 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:33.802220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:33.804330 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:33.807525 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:33.807534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:33.807549 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:33.808117 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:33.808467 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:33.808470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:33.808473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:33.808639 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:33.809217 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:33.809224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:33.809535 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 05:07:33.809761 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 05:07:33.810155 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":1382,"threads_started":1}
I20260925 05:07:33.812561 15148 heartbeater.cc:347] Connected to a master server at 127.8.215.62:39277
I20260925 05:07:33.812896 15148 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:33.813578 15148 heartbeater.cc:511] Master 127.8.215.62:39277 requested a full tablet report, sending...
I20260925 05:07:33.815049 14692 ts_manager.cc:194] Registered new tserver with Master: 517c43724ea44afaa47c74c79cc1d80b (127.8.215.3:35569)
I20260925 05:07:33.815788 14692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:33519
I20260925 05:07:33.820255 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:33.822186 14881 heartbeater.cc:503] Master 127.8.215.62:39277 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/test-admin.keytab.
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.851142 9052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 05:07:33.857322 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:33.854291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.8.215.1:34491)
0925 05:07:33.855021 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.855035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.855079 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:07:33.855147 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.855154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.855438 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.855679 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.856806 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35124: BlockingRecv error: recv got EOF from 127.0.0.1:35124 (error 108)
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":340,"threads_started":1}
W20260925 05:07:33.857285 9052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.858877 9052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.860329 9052 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:33.875141 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.862088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.8.215.1:34491)
0925 05:07:33.862280 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.862286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.862515 (+ 229us) server_negotiation.cc:407] Connection header received
0925 05:07:33.862656 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.862660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.862714 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.862790 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.863681 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.864609 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.865854 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.866398 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.866498 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.870136 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:33.870146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:33.870161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:33.870190 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:33.872837 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.873415 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.873421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.873424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.873534 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.873855 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.873858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.873860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.874187 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:33.874386 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:33.874703 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 05:07:33.874889 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 05:07:33.881733 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:33.879392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.8.215.1:34491)
0925 05:07:33.879649 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.879658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.879936 (+ 278us) server_negotiation.cc:407] Connection header received
0925 05:07:33.880165 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.880183 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.880266 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.880386 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.881422 (+ 1036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35148: BlockingRecv error: recv got EOF from 127.0.0.1:35148 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 05:07:33.881865 9052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.883543 9052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.885141 9052 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:33.899497 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.886557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.8.215.1:34491)
0925 05:07:33.886749 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.886756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.886995 (+ 239us) server_negotiation.cc:407] Connection header received
0925 05:07:33.887128 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.887131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.887183 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.887269 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.888495 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.889392 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.890848 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.891489 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.891613 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.894918 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:33.894923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:33.894926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:33.894967 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:33.897577 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.898132 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.898136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.898138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.898247 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.898556 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.898568 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.898570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.898792 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:33.898971 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:33.899153 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:07:33.899288 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 05:07:33.903723 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:33.902097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.8.215.1:34491)
0925 05:07:33.902289 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.902294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.902488 (+ 194us) server_negotiation.cc:407] Connection header received
0925 05:07:33.902653 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.902658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.902734 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.902831 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.903510 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35178: BlockingRecv error: recv got EOF from 127.0.0.1:35178 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:07:33.903868 9052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312801912195-9052-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.905292 9052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:33.906504 9052 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:07:33 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:33.920095 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.907580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.8.215.1:34491)
0925 05:07:33.907784 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.907790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.908032 (+ 242us) server_negotiation.cc:407] Connection header received
0925 05:07:33.908206 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.908211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.908269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.908366 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.909327 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.910013 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.911240 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.911642 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.911763 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.915283 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:33.915295 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:33.915299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:33.915325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:33.917654 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.918263 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.918270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.918279 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.918415 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.918777 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.918782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.918786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.919022 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:33.919348 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:33.919584 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 05:07:33.919780 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:07:33.921814 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14753
I20260925 05:07:33.933235 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:33.922564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.8.215.1:34491)
0925 05:07:33.922789 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:33.922795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:33.922977 (+ 182us) server_negotiation.cc:407] Connection header received
0925 05:07:33.923090 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:33.923093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:33.923144 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:33.923212 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:33.924030 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.924663 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.925853 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:33.926249 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:33.926336 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:33.928285 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:33.928289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:33.928292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:33.928317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:33.930810 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.931558 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.931561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.931564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.931672 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:33.932019 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:33.932022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:33.932024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:33.932250 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:33.932426 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:33.932672 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 05:07:33.932886 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 05:07:33.933822 14876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:34.016515 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14753
I20260925 05:07:34.033694 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14887
Sep 25 05:07:34 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:07:34.050024 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:34.034672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.8.215.2:39961)
0925 05:07:34.035239 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:34.035253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:34.035280 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:07:34.035340 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:34.035346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:34.035593 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:34.035722 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:34.036623 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:34.037854 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:34.039119 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:34.039829 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:34.039965 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:34.043810 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:34.043822 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:34.043837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:34.043869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:34.047212 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:34.047881 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:34.047890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:34.047894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:34.048018 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:34.048411 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:34.048417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:34.048421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:34.048805 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:34.049125 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:34.049463 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 05:07:34.049705 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":311,"threads_started":1}
I20260925 05:07:34.051178 15010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:34.132207 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14887
I20260925 05:07:34.148515 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15020
Sep 25 05:07:34 dist-test-slave-wpdm krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:07:34.163359 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:34.149473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.8.215.3:35569)
0925 05:07:34.150094 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:34.150100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:34.150128 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:07:34.150188 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:34.150195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:34.150397 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:34.150532 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:34.151483 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:34.152685 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:34.153916 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:34.154464 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:34.154566 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:34.158200 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:34.158209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:34.158223 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:34.158256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:34.160910 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:34.161541 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:34.161547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:34.161549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:34.161669 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:34.162068 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:34.162072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:34.162074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:34.162381 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:34.162668 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:34.162890 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 05:07:34.163083 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":318,"threads_started":1}
I20260925 05:07:34.164606 15143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:34.246654 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15020
I20260925 05:07:34.262730 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 14660
I20260925 05:07:34.263909 14722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:34.363785 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 14660
2026-09-25T05:07:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3881 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:34 dist-test-slave-wpdm krb5kdc[15175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:34 dist-test-slave-wpdm krb5kdc[15175](info): set up 2 sockets
Sep 25 05:07:34 dist-test-slave-wpdm krb5kdc[15175](info): commencing operation
krb5kdc: starting...
W20260925 05:07:35.478542 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:35 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab.
Entry for principal oryx/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab.
I20260925 05:07:35.692873 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:36215
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab
--principal=oryx/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:36.013098 15188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:36.013379 15188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:36.013558 15188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:36.020778 15188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:36.020931 15188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:36.021015 15188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:36.021086 15188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:36.021170 15188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:36.030913 15188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab
--principal=oryx/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:36215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:36.032263 15188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:36.033619 15188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:36.045538 15196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:36.045459 15194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:36.045578 15193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:36.046254 15188 server_base.cc:1034] running on GCE node
I20260925 05:07:36.047098 15188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:36.048512 15188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:36.049710 15188 hybrid_clock.cc:648] HybridClock initialized: now 1790312856049665 us; error 34 us; skew 500 ppm
I20260925 05:07:36.050086 15188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:36 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312856, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:36.058616 15188 init.cc:377] Logged in from keytab as oryx/127.8.215.62@KRBTEST.COM (short username oryx)
I20260925 05:07:36.060098 15188 webserver.cc:460] Webserver started at http://127.8.215.62:33495/ using document root <none> and password file <none>
I20260925 05:07:36.060874 15188 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:36.061012 15188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:36.061318 15188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:36.064231 15188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "6d648d6d50cd4c9bb2089acea54b0f6f"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "2d3f6eba20cb76a352a88be2c728a2c1"
server_key_iv: "152e31eb18b2a17834b6db50c8006dbb"
server_key_version: "encryptionkey@0"
I20260925 05:07:36.065060 15188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "6d648d6d50cd4c9bb2089acea54b0f6f"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "2d3f6eba20cb76a352a88be2c728a2c1"
server_key_iv: "152e31eb18b2a17834b6db50c8006dbb"
server_key_version: "encryptionkey@0"
I20260925 05:07:36.070158 15188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:07:36.073493 15203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:36.074779 15188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:36.074975 15188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "6d648d6d50cd4c9bb2089acea54b0f6f"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "2d3f6eba20cb76a352a88be2c728a2c1"
server_key_iv: "152e31eb18b2a17834b6db50c8006dbb"
server_key_version: "encryptionkey@0"
I20260925 05:07:36.075160 15188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:36.101361 15188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:36.105957 15188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:36.106264 15188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:36.123495 15188 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:36215
I20260925 05:07:36.123528 15254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:36215 every 8 connection(s)
I20260925 05:07:36.124852 15188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:36.128417 15255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:36.133225 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15188
I20260925 05:07:36.133445 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:36.134238 9052 external_mini_cluster.cc:1428] Setting key 071544900ae15c897882a1c8ed0288eb
I20260925 05:07:36.139629 15255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Bootstrap starting.
I20260925 05:07:36.143286 15255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:36.144527 15255 log.cc:824] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:36.147996 15255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: No bootstrap required, opened a new log
Sep 25 05:07:36 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:36.155690 15255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } }
I20260925 05:07:36.156067 15255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:36.156205 15255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d648d6d50cd4c9bb2089acea54b0f6f, State: Initialized, Role: FOLLOWER
I20260925 05:07:36.156948 15255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } }
I20260925 05:07:36.157258 15255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:36.157410 15255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:36.157605 15255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:36.158371 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:36.136346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.8.215.62:36215)
0925 05:07:36.137151 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:36.137162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:36.137204 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:07:36.138804 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:36.138825 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:36.139191 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:36.139498 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:36.141261 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:36.142604 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:36.144398 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:36.145392 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:36.145615 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:36.150240 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:36.150250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:36.150266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:36.150312 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:36.153846 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:36.154610 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:36.154628 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:36.154634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:36.154826 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:36.155262 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:36.155269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:36.155273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:36.155818 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:36.156155 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:36.156636 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 05:07:36.157088 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":395,"threads_started":1}
I20260925 05:07:36.160384 15255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } }
I20260925 05:07:36.160962 15255 leader_election.cc:302] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d648d6d50cd4c9bb2089acea54b0f6f; no voters:
I20260925 05:07:36.161805 15255 leader_election.cc:288] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:36.161916 15260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:36.162672 15260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 LEADER]: Becoming Leader. State: Replica: 6d648d6d50cd4c9bb2089acea54b0f6f, State: Running, Role: LEADER
I20260925 05:07:36.163245 15260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } }
I20260925 05:07:36.163790 15255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:36.165947 15261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } }
I20260925 05:07:36.166357 15261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:36.166361 15262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d648d6d50cd4c9bb2089acea54b0f6f. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } }
I20260925 05:07:36.166901 15262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:36.169103 15269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:36.174530 15269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:36.183214 15269 catalog_manager.cc:1347] Generated new cluster ID: dfbfdca724914b0baa3b67d52b054438
I20260925 05:07:36.183342 15269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:36.194754 15269 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:36.196012 15269 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:36.202428 15269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Generated new TSK 0
I20260925 05:07:36.203372 15269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab.
Entry for principal oryx/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab.
I20260925 05:07:36.260993 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:36215
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:36.587237 15281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:36.587549 15281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:36.587716 15281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:36.594923 15281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:36.595091 15281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:36.595273 15281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:36.604941 15281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:36215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:36.606344 15281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:36.607673 15281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:36.620654 15286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:36.620704 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:36.620832 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:36.621668 15281 server_base.cc:1034] running on GCE node
I20260925 05:07:36.622342 15281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:36.623237 15281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:36.624461 15281 hybrid_clock.cc:648] HybridClock initialized: now 1790312856624412 us; error 68 us; skew 500 ppm
I20260925 05:07:36.624821 15281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:36 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312856, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:36.627837 15281 init.cc:377] Logged in from keytab as oryx/127.8.215.1@KRBTEST.COM (short username oryx)
I20260925 05:07:36.629340 15281 webserver.cc:460] Webserver started at http://127.8.215.1:44339/ using document root <none> and password file <none>
I20260925 05:07:36.630025 15281 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:36.630172 15281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:36.630493 15281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:36.633453 15281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "d2cf57f6810e4f278a02eeae532b7258"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "27343f3439e7612c80b3147f0ce98631"
server_key_iv: "5724343ccab6479470474ee494a21fe2"
server_key_version: "encryptionkey@0"
I20260925 05:07:36.634219 15281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "d2cf57f6810e4f278a02eeae532b7258"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "27343f3439e7612c80b3147f0ce98631"
server_key_iv: "5724343ccab6479470474ee494a21fe2"
server_key_version: "encryptionkey@0"
I20260925 05:07:36.639256 15281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:36.642563 15296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:36.643918 15281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:36.644109 15281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "d2cf57f6810e4f278a02eeae532b7258"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "27343f3439e7612c80b3147f0ce98631"
server_key_iv: "5724343ccab6479470474ee494a21fe2"
server_key_version: "encryptionkey@0"
I20260925 05:07:36.644315 15281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:36.681905 15281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:36.686076 15281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:36.686370 15281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:36.687345 15281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:36.688678 15281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:36.688808 15281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 05:07:36.688957 15281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:36.689031 15281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:36.722517 15281 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:39177
I20260925 05:07:36.722535 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:39177 every 8 connection(s)
I20260925 05:07:36.724081 15281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:36.734508 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15281
I20260925 05:07:36.734736 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:36.735352 9052 external_mini_cluster.cc:1428] Setting key 0d1e151e13cd4b06aa993e5526c3ac1b
Sep 25 05:07:36 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312856, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:36.743784 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:36.726642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:46519 (local address 127.8.215.62:36215)
0925 05:07:36.727075 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:36.727080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:36.728014 (+ 934us) server_negotiation.cc:407] Connection header received
0925 05:07:36.729900 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:36.729905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:36.729991 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:36.730108 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:36.731552 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:36.732263 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:36.733862 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:36.734270 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:36.734437 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:36.738183 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:36.738189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:36.738195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:36.738232 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:36.740786 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:36.741500 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:36.741505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:36.741509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:36.741645 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:36.742072 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:36.742077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:36.742080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:36.742343 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:36.742499 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:36.743193 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 05:07:36.743406 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260925 05:07:36.744776 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:36.727202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:46519)
0925 05:07:36.727847 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:36.727897 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:36.729618 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:36.730337 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:36.730345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:36.730836 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:36.731382 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:36.731403 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:36.732422 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:36.732430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:36.733711 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:36.733722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:36.734542 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:36.734549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:36.734845 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:36.735542 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:36.735565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:36.737960 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:36.740914 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:36.740923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:36.740947 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:36.741371 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:36.741764 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:36.741769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:36.741778 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:36.741951 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:36.742473 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:36.742481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:36.742820 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 05:07:36.743106 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 05:07:36.743644 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":256,"threads_started":1}
I20260925 05:07:36.746289 15409 heartbeater.cc:347] Connected to a master server at 127.8.215.62:36215
I20260925 05:07:36.746644 15409 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:36.747565 15409 heartbeater.cc:511] Master 127.8.215.62:36215 requested a full tablet report, sending...
I20260925 05:07:36.749840 15220 ts_manager.cc:194] Registered new tserver with Master: d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177)
I20260925 05:07:36.751230 15220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:46519
WARNING: no policy specified for oryx/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab.
Entry for principal oryx/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab.
I20260925 05:07:36.783963 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:36215
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:37.107402 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:37.107698 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:37.107893 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:37.115195 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:37.115306 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:37.115504 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:37.125640 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:36215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:37.126912 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:37.128247 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:37.141290 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:37.141259 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:37.141690 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:37.142251 15415 server_base.cc:1034] running on GCE node
I20260925 05:07:37.142822 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:37.143728 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:37.144928 15415 hybrid_clock.cc:648] HybridClock initialized: now 1790312857144862 us; error 55 us; skew 500 ppm
I20260925 05:07:37.145218 15415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:37 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312857, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:37.148445 15415 init.cc:377] Logged in from keytab as oryx/127.8.215.2@KRBTEST.COM (short username oryx)
I20260925 05:07:37.149972 15415 webserver.cc:460] Webserver started at http://127.8.215.2:39883/ using document root <none> and password file <none>
I20260925 05:07:37.150698 15415 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:37.150828 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:37.151151 15415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:37.154116 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "5cc7a29b523b433b8056553dd4a77ae0"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "b51d713a72947b2ef50034a6e7bd697b"
server_key_iv: "13151834231c4b1854e314239c99284c"
server_key_version: "encryptionkey@0"
I20260925 05:07:37.154857 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc7a29b523b433b8056553dd4a77ae0"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "b51d713a72947b2ef50034a6e7bd697b"
server_key_iv: "13151834231c4b1854e314239c99284c"
server_key_version: "encryptionkey@0"
I20260925 05:07:37.159816 15415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:07:37.163232 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:37.164875 15415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:37.165077 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "5cc7a29b523b433b8056553dd4a77ae0"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "b51d713a72947b2ef50034a6e7bd697b"
server_key_iv: "13151834231c4b1854e314239c99284c"
server_key_version: "encryptionkey@0"
I20260925 05:07:37.165292 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:37.182592 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:37.186599 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:37.186883 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:37.187804 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:37.189141 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:37.189211 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:37.189357 15415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:37.189415 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:37.221459 15415 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:34959
I20260925 05:07:37.221549 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:34959 every 8 connection(s)
I20260925 05:07:37.222734 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:37.223500 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15415
I20260925 05:07:37.223711 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:37.224280 9052 external_mini_cluster.cc:1428] Setting key 9f375b1058be5104df2a1e8ccd974351
Sep 25 05:07:37 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312857, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:37.245671 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:37.225544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:37953 (local address 127.8.215.62:36215)
0925 05:07:37.225742 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:37.225748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:37.227163 (+ 1415us) server_negotiation.cc:407] Connection header received
0925 05:07:37.229932 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:37.229935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:37.230049 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:37.230150 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:37.232202 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.232943 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.234946 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.235363 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.235496 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:37.239539 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:37.239543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:37.239547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:37.239583 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:37.242181 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:37.243138 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:37.243144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:37.243147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:37.243278 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:37.243771 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:37.243774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:37.243776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:37.244029 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:37.244165 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:37.245183 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0925 05:07:37.245348 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:07:37.247185 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:37.226162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:37953)
0925 05:07:37.226954 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:37.227004 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:37.229655 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:37.230413 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:37.230427 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:37.231221 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:37.232020 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:37.232048 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.233093 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.233101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:37.234791 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:37.234803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.235629 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.235638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:37.235871 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:37.236587 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:37.236613 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:37.239329 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:37.242313 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:37.242324 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:37.242345 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:37.243008 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:37.243392 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:37.243400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:37.243404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:37.243661 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:37.244189 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:37.244199 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:37.244748 (+ 549us) client_negotiation.cc:769] Sending connection context
0925 05:07:37.245102 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 05:07:37.245687 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":265,"threads_started":1}
WARNING: no policy specified for oryx/127.8.215.3@KRBTEST.COM; defaulting to no policy
I20260925 05:07:37.248677 15543 heartbeater.cc:347] Connected to a master server at 127.8.215.62:36215
I20260925 05:07:37.249074 15543 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:37.249830 15543 heartbeater.cc:511] Master 127.8.215.62:36215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.3@KRBTEST.COM" created.
I20260925 05:07:37.251421 15220 ts_manager.cc:194] Registered new tserver with Master: 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959)
I20260925 05:07:37.252662 15220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:37953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab.
Entry for principal oryx/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab.
I20260925 05:07:37.276687 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:36215
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:37.602717 15548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:37.603057 15548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:37.603241 15548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:37.610459 15548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:37.610615 15548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:37.610805 15548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:37.620380 15548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:36215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:37.621603 15548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:37.622816 15548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:37.635587 15553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:37.635476 15556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:37.635504 15554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:37.636482 15548 server_base.cc:1034] running on GCE node
I20260925 05:07:37.637107 15548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:37.638019 15548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:37.639245 15548 hybrid_clock.cc:648] HybridClock initialized: now 1790312857639198 us; error 40 us; skew 500 ppm
I20260925 05:07:37.639598 15548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:37 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312857, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:37.642740 15548 init.cc:377] Logged in from keytab as oryx/127.8.215.3@KRBTEST.COM (short username oryx)
I20260925 05:07:37.644265 15548 webserver.cc:460] Webserver started at http://127.8.215.3:38865/ using document root <none> and password file <none>
I20260925 05:07:37.645037 15548 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:37.645172 15548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:37.645480 15548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:37.648520 15548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "3ce603504d2b4ea1902c513b97272c52"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "a5af1f2aaa958cbf429fd59bff26822b"
server_key_iv: "df8b240d4f66d5dfcea04156c8fb8122"
server_key_version: "encryptionkey@0"
I20260925 05:07:37.649330 15548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "3ce603504d2b4ea1902c513b97272c52"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "a5af1f2aaa958cbf429fd59bff26822b"
server_key_iv: "df8b240d4f66d5dfcea04156c8fb8122"
server_key_version: "encryptionkey@0"
I20260925 05:07:37.654371 15548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:37.657781 15563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:37.659083 15548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:37.659250 15548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "3ce603504d2b4ea1902c513b97272c52"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "a5af1f2aaa958cbf429fd59bff26822b"
server_key_iv: "df8b240d4f66d5dfcea04156c8fb8122"
server_key_version: "encryptionkey@0"
I20260925 05:07:37.659446 15548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:37.676268 15548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:37.680316 15548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:37.680609 15548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:37.681515 15548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:37.682801 15548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:37.682870 15548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:37.683015 15548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:37.683064 15548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:37.715242 15548 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39793
I20260925 05:07:37.715322 15675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39793 every 8 connection(s)
I20260925 05:07:37.716572 15548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:37.726893 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15548
I20260925 05:07:37.727075 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:37.727576 9052 external_mini_cluster.cc:1428] Setting key 8f85350080bfa69568b5ffb1d50ca801
Sep 25 05:07:37 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312857, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:37.736122 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:37.719324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:48283 (local address 127.8.215.62:36215)
0925 05:07:37.719514 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:37.719519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:37.720691 (+ 1172us) server_negotiation.cc:407] Connection header received
0925 05:07:37.722558 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:37.722564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:37.722683 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:37.722797 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:37.724268 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.725113 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.726834 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.727314 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.727420 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:37.731067 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:37.731071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:37.731075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:37.731104 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:37.733561 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:37.734230 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:37.734233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:37.734236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:37.734349 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:37.734692 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:37.734695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:37.734697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:37.734939 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:37.735045 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:37.735650 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 05:07:37.735793 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:07:37.737030 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:37.719811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:48283)
0925 05:07:37.720525 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:37.720568 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:37.722299 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:37.723033 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:37.723041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:37.723550 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:37.724099 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:37.724115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.725271 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.725276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:37.726687 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:37.726702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:37.727531 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:37.727539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:37.727825 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:37.728562 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:37.728586 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:37.730878 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:37.733690 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:37.733696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:37.733708 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:37.734129 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:37.734439 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:37.734442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:37.734444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:37.734606 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:37.735039 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:37.735044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:37.735334 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 05:07:37.735565 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 05:07:37.735918 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":271,"threads_started":1}
I20260925 05:07:37.738360 15676 heartbeater.cc:347] Connected to a master server at 127.8.215.62:36215
I20260925 05:07:37.738660 15676 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:37.739349 15676 heartbeater.cc:511] Master 127.8.215.62:36215 requested a full tablet report, sending...
I20260925 05:07:37.740902 15220 ts_manager.cc:194] Registered new tserver with Master: 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:37.741950 15220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:48283
I20260925 05:07:37.742797 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:37.753981 15409 heartbeater.cc:503] Master 127.8.215.62:36215 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.8.215.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.61.keytab.
Entry for principal oryx/127.8.215.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.61.keytab.
I20260925 05:07:37.791685 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:43439
--webserver_interface=127.8.215.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.61.keytab
--principal=oryx/127.8.215.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36215,127.8.215.61:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:38.112891 15681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:38.113291 15681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:38.113385 15681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:38.120324 15681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:38.120419 15681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:38.120465 15681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:38.120517 15681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:38.120558 15681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:38.134773 15681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36215,127.8.215.61:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.61.keytab
--principal=oryx/127.8.215.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.215.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
W20260925 05:07:38.135169 15681 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.215.62:36215,127.8.215.61:43439'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 05:07:38.136291 15681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:38.137634 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:38.149188 15686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:38.149125 15687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:38.149132 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:38.149976 15681 server_base.cc:1034] running on GCE node
I20260925 05:07:38.150759 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:38.152104 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:38.153335 15681 hybrid_clock.cc:648] HybridClock initialized: now 1790312858153266 us; error 44 us; skew 500 ppm
I20260925 05:07:38.153693 15681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:38.156802 15681 init.cc:377] Logged in from keytab as oryx/127.8.215.61@KRBTEST.COM (short username oryx)
I20260925 05:07:38.158254 15681 webserver.cc:460] Webserver started at http://127.8.215.61:40699/ using document root <none> and password file <none>
I20260925 05:07:38.158913 15681 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:38.159039 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:38.159328 15681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:38.162240 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/instance:
uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "a16ff8a6e78e97b4adb297ab27d60174"
server_key_iv: "a9d8da25c0d3778f83a0c1d58c698590"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.163017 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal/instance:
uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "a16ff8a6e78e97b4adb297ab27d60174"
server_key_iv: "a9d8da25c0d3778f83a0c1d58c698590"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.168203 15681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:07:38.171646 15696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:38.172937 15681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:07:38.173103 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "a16ff8a6e78e97b4adb297ab27d60174"
server_key_iv: "a9d8da25c0d3778f83a0c1d58c698590"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.173309 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:38.192946 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:38.196625 15681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:38.196931 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:38.213825 15681 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.61:43439
I20260925 05:07:38.213855 15747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.61:43439 every 8 connection(s)
I20260925 05:07:38.215127 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
I20260925 05:07:38.218770 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:38.220584 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15681
I20260925 05:07:38.229038 15748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.61@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
I20260925 05:07:38.251389 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.232580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.8.215.62:36215)
0925 05:07:38.232789 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.232794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.233406 (+ 612us) server_negotiation.cc:407] Connection header received
0925 05:07:38.233739 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.233745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.233818 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.233925 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.235657 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.237033 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.239242 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.239941 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.240081 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.244338 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.244343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.244346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.244379 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.247795 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.248846 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.248853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.248856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.248994 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.249529 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.249534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.249539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.249974 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.250129 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.250780 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.251033 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:07:38.252488 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.232545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:41806)
0925 05:07:38.233247 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.233284 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.233549 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.234033 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.234044 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.234894 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.235485 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.235501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.237161 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.237165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.239066 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.239081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.240208 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.240216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.240401 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.241290 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.241327 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.244158 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.247985 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.247994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.248003 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.248684 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.249154 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.249161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.249167 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.249359 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.250100 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.250106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.250358 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.250689 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.251226 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":23,"thread_start_us":381,"threads_started":1}
I20260925 05:07:38.252467 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.222821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.8.215.61:43439)
0925 05:07:38.230425 (+ 7604us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.230437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.230477 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:07:38.232428 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.232463 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.233045 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.233436 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.235369 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.237316 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.239123 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.240081 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.240273 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.245070 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.245078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.245090 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.245120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.248527 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.249220 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.249226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.249228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.249367 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.249761 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.249765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.249769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.250377 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.250597 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.250999 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.251354 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":7388,"spinlock_wait_cycles":38656,"thread_start_us":7168,"threads_started":1}
I20260925 05:07:38.254359 15748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:07:38.255221 15543 heartbeater.cc:503] Master 127.8.215.62:36215 was elected leader, sending a full tablet report...
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.61@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
I20260925 05:07:38.266747 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.255270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37584)
0925 05:07:38.255725 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.255743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.255871 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.256170 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.256175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.256546 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.256665 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.256672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.257702 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.257707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.258603 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.258614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.259275 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.259281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.259491 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.259971 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.259986 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.261955 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.264779 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.264783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.264790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.265212 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.265504 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.265507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.265509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.265630 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.266088 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.266092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.266210 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.266312 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.266505 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":4}
I20260925 05:07:38.266932 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.255563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.8.215.61:43439)
0925 05:07:38.255717 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.255722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.255812 (+ 90us) server_negotiation.cc:407] Connection header received
0925 05:07:38.255993 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.255998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.256061 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.256156 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.256831 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.257561 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.258737 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.259140 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.259276 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.262109 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.262113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.262117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.262142 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.264625 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.265305 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.265308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.265311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.265414 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.265725 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.265727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.265729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.265994 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.266101 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.266401 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.266609 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":35}
I20260925 05:07:38.271540 15748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Bootstrap starting.
I20260925 05:07:38.274573 15748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:38.275550 15748 log.cc:824] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:38.278314 15748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: No bootstrap required, opened a new log
I20260925 05:07:38.285126 15748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } }
I20260925 05:07:38.285405 15748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:38.285482 15748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aafd2c3c69c4d0897dc79edf79fdd3b, State: Initialized, Role: FOLLOWER
I20260925 05:07:38.286019 15748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } }
I20260925 05:07:38.286755 15754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } } }
I20260925 05:07:38.287076 15754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:38.287693 15748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:38.290215 15754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:38.290462 15754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } }
I20260925 05:07:38.291905 15754 leader_election.cc:288] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215)
I20260925 05:07:38.292799 15230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" is_pre_election: true
I20260925 05:07:38.293913 15699 leader_election.cc:302] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2aafd2c3c69c4d0897dc79edf79fdd3b; no voters: 6d648d6d50cd4c9bb2089acea54b0f6f
I20260925 05:07:38.294312 15754 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 05:07:38.297458 15765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:07:38.297575 15765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:07:38.298273 15681 master_runner.cc:428] Detected that this master 2aafd2c3c69c4d0897dc79edf79fdd3b is joining an existing cluster
I20260925 05:07:38.298358 15681 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.215.61:43439
I20260925 05:07:38.321105 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.302208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.8.215.62:36215)
0925 05:07:38.302381 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.302389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.303326 (+ 937us) server_negotiation.cc:407] Connection header received
0925 05:07:38.303575 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.303581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.303681 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.303791 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.305383 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.306465 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.308657 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.309341 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.309443 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.312604 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.312610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.312614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.312650 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.316706 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.318869 (+ 2163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.318874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.318878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.319013 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.319490 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.319495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.319498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.319820 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.319938 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.320525 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.320760 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:07:38.321162 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.302469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37600)
0925 05:07:38.303528 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.303546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.303672 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.304056 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.304059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.304661 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.305175 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.305187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.306617 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.306621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.308506 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.308519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.309753 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.309759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.310081 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.310993 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.311013 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.312403 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.318007 (+ 5604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.318015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.318022 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.318732 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.319159 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.319163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.319167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.319357 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.320023 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.320027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.320218 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.320371 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.320661 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":312,"spinlock_wait_cycles":9472,"thread_start_us":392,"threads_started":1}
I20260925 05:07:38.322146 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.302302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.8.215.61:43439)
0925 05:07:38.302484 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.302493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.303637 (+ 1144us) server_negotiation.cc:407] Connection header received
0925 05:07:38.303784 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.303789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.303854 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.303948 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.305300 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.306379 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.308646 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.309371 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.309450 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.312608 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.312614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.312618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.312649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.317878 (+ 5229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.318875 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.318884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.318889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.319041 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.319474 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.319482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.319485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.319820 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.319960 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.321580 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.321781 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:07:38.322736 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.302184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:41820)
0925 05:07:38.303219 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.303243 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.303435 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.303846 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.303851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.304672 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.305207 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.305219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.306621 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.306625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.308506 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.308519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.309728 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.309739 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.310085 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.310924 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.310954 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.312403 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.318072 (+ 5669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.318075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.318081 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.318732 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.319159 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.319163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.319167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.319355 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.320024 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.320027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.320239 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.321795 (+ 1556us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.322071 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":258,"spinlock_wait_cycles":14080,"thread_start_us":1042,"threads_started":1}
WARNING: no policy specified for oryx/127.8.215.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.60@KRBTEST.COM" created.
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312856, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.62@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
I20260925 05:07:38.345762 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.328443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.8.215.61:43439)
0925 05:07:38.328616 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.328625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.329290 (+ 665us) server_negotiation.cc:407] Connection header received
0925 05:07:38.329607 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.329613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.329731 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.329842 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.330815 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.331529 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.332805 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.333272 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.333401 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.338330 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.338339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.338347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.338392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.342430 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.343414 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.343420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.343424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.343599 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.344069 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.344081 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.344087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.344418 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.344593 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.345159 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.345393 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:07:38.345816 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.328416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37606)
0925 05:07:38.329174 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.329208 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.329407 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.329893 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.329900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.330505 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.330681 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.330688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.331654 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.331661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.332636 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.332647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.333442 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.333448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.333665 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.334331 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.334351 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.338108 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.342618 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.342634 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.342643 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.343253 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.343740 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.343745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.343749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.343916 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.344572 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.344581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.344856 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.345055 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.345393 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":372,"threads_started":1}
I20260925 05:07:38.347776 15219 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.215.61:43439: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b"
member_type: NON_VOTER
last_known_addr {
host: "127.8.215.61"
port: 43439
}
attrs {
promote: true
}
}
dest_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f"
cas_config_opid_index: -1
I20260925 05:07:38.348915 15219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } }
I20260925 05:07:38.357700 15777 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2aafd2c3c69c4d0897dc79edf79fdd3b (127.8.215.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } }
I20260925 05:07:38.358880 15723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:38.363108 15723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 05:07:38.363972 15777 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.215.61:43439
I20260925 05:07:38.364959 15681 master.cc:537] Master@127.8.215.61:43439 shutting down...
I20260925 05:07:38.364681 15778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } }
I20260925 05:07:38.365206 15778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:38.370899 15777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.60.keytab.
Entry for principal oryx/127.8.215.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.60.keytab.
I20260925 05:07:38.373739 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:41459
--webserver_interface=127.8.215.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.60.keytab
--principal=oryx/127.8.215.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:38.377219 15204 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 05:07:38.378816 15204 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f -> Peer 2aafd2c3c69c4d0897dc79edf79fdd3b (127.8.215.61:43439): Couldn't send request to peer 2aafd2c3c69c4d0897dc79edf79fdd3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 05:07:38.381955 15681 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:38.382370 15681 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:38.382601 15681 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: stopping tablet replica
I20260925 05:07:38.388777 15681 master.cc:552] Master@127.8.215.61:43439 shutdown complete.
I20260925 05:07:38.388861 15681 master_runner.cc:305] Clearing existing system tablet
I20260925 05:07:38.389281 15681 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:38.389410 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:38.391924 15681 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 05:07:38.393365 15784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:38.393661 15681 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:38.393947 15681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:38.394029 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "a16ff8a6e78e97b4adb297ab27d60174"
server_key_iv: "a9d8da25c0d3778f83a0c1d58c698590"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.397233 15681 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:38.401940 15681 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:07:38.402117 15681 log.cc:1196] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 05:07:38.402493 15681 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Deleting consensus metadata
I20260925 05:07:38.402736 15681 master_runner.cc:315] Copying system tablet from 127.8.215.62:36215
I20260925 05:07:38.405016 15681 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.215.62:36215
I20260925 05:07:38.424185 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.405767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.8.215.62:36215)
0925 05:07:38.405972 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.405980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.413265 (+ 7285us) server_negotiation.cc:407] Connection header received
0925 05:07:38.413472 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.413480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.413575 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.413694 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.414533 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.415234 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.416416 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.416881 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.417031 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.418997 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.419009 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.419013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.419044 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.421653 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.422263 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.422271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.422274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.422396 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.422751 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.422756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.422760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.423044 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.423204 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.423597 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.423814 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:38.424342 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.405729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:41824)
0925 05:07:38.413128 (+ 7399us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.413148 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.413327 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.413721 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.413726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.414158 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.414357 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.414366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.415368 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.415375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.416268 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.416279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.417046 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.417052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.417262 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.417898 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.417915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.418807 (+ 892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.421785 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.421791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.421795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.422145 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.422510 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.422514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.422518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.422646 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.423159 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.423164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.423347 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.423495 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.423896 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7248,"thread_start_us":7168,"threads_started":1}
I20260925 05:07:38.425351 15240 tablet_copy_service.cc:140] P 6d648d6d50cd4c9bb2089acea54b0f6f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2aafd2c3c69c4d0897dc79edf79fdd3b ({username='oryx', principal='oryx/127.8.215.61@KRBTEST.COM'} at 127.0.0.1:41824)
I20260925 05:07:38.425577 15240 tablet_copy_service.cc:161] P 6d648d6d50cd4c9bb2089acea54b0f6f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2aafd2c3c69c4d0897dc79edf79fdd3b at {username='oryx', principal='oryx/127.8.215.61@KRBTEST.COM'} at 127.0.0.1:41824: session id = 2aafd2c3c69c4d0897dc79edf79fdd3b-00000000000000000000000000000000
I20260925 05:07:38.428504 15240 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:07:38.430398 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:07:38.435113 15681 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: tablet copy: Starting download of 0 data blocks...
I20260925 05:07:38.435403 15681 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: tablet copy: Starting download of 1 WAL segments...
I20260925 05:07:38.437719 15681 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:07:38.440673 15240 tablet_copy_service.cc:342] P 6d648d6d50cd4c9bb2089acea54b0f6f: Request end of tablet copy session 2aafd2c3c69c4d0897dc79edf79fdd3b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.215.61@KRBTEST.COM'} at 127.0.0.1:41824
I20260925 05:07:38.440886 15240 tablet_copy_service.cc:433] P 6d648d6d50cd4c9bb2089acea54b0f6f: ending tablet copy session 2aafd2c3c69c4d0897dc79edf79fdd3b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2aafd2c3c69c4d0897dc79edf79fdd3b
I20260925 05:07:38.453121 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:38.456053 15791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:38.456691 15792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:38.464151 15794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:38.468698 15681 server_base.cc:1034] running on GCE node
I20260925 05:07:38.469179 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:38.469910 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:38.471192 15681 hybrid_clock.cc:648] HybridClock initialized: now 1790312858471133 us; error 36 us; skew 500 ppm
I20260925 05:07:38.471422 15681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:38.473964 15681 init.cc:377] Logged in from keytab as oryx/127.8.215.61@KRBTEST.COM (short username oryx)
I20260925 05:07:38.477499 15681 webserver.cc:460] Webserver started at http://127.8.215.61:45623/ using document root <none> and password file <none>
I20260925 05:07:38.477993 15681 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:38.478085 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:38.481825 15681 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 05:07:38.484252 15801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:38.484843 15681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:07:38.485034 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "a16ff8a6e78e97b4adb297ab27d60174"
server_key_iv: "a9d8da25c0d3778f83a0c1d58c698590"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.485283 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:38.536103 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:38.540606 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:38.561731 15681 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.61:43439
I20260925 05:07:38.561769 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.61:43439 every 8 connection(s)
I20260925 05:07:38.563169 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
I20260925 05:07:38.568754 15853 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:07:38.572717 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Bootstrap starting.
I20260925 05:07:38.588541 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:07:38.589074 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Bootstrap complete.
I20260925 05:07:38.590188 15853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } }
I20260925 05:07:38.590445 15853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2aafd2c3c69c4d0897dc79edf79fdd3b, State: Initialized, Role: LEARNER
I20260925 05:07:38.590747 15853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } }
I20260925 05:07:38.591585 15856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } }
I20260925 05:07:38.591825 15853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:38.591990 15856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: LEARNER
I20260925 05:07:38.602334 15867 catalog_manager.cc:1259] Loaded cluster ID: dfbfdca724914b0baa3b67d52b054438
I20260925 05:07:38.602466 15867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: loading cluster ID for follower catalog manager: success
I20260925 05:07:38.606853 15867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: acquiring CA information for follower catalog manager: success
I20260925 05:07:38.609505 15867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:07:38.744992 15676 heartbeater.cc:503] Master 127.8.215.62:36215 was elected leader, sending a full tablet report...
W20260925 05:07:38.781529 15782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:38.781868 15782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:38.782033 15782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:38.789906 15782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:38.790047 15782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:38.790163 15782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:38.790261 15782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:38.790338 15782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:38.800949 15782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.60.keytab
--principal=oryx/127.8.215.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.215.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:38.802263 15782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:38.803539 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:38.815250 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:38.815336 15875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:38.815358 15872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:38.816227 15782 server_base.cc:1034] running on GCE node
I20260925 05:07:38.816962 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:38.818282 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:38.819514 15782 hybrid_clock.cc:648] HybridClock initialized: now 1790312858819464 us; error 61 us; skew 500 ppm
I20260925 05:07:38.819850 15782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:38.822934 15782 init.cc:377] Logged in from keytab as oryx/127.8.215.60@KRBTEST.COM (short username oryx)
I20260925 05:07:38.824270 15782 webserver.cc:460] Webserver started at http://127.8.215.60:45649/ using document root <none> and password file <none>
I20260925 05:07:38.825024 15782 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:38.825167 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:38.825459 15782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:38.828332 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/instance:
uuid: "22d428d011a842c693e4d23d7f4addcf"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "e24f8036fc4b4be12fd616a3311e3a2b"
server_key_iv: "c0a34ddc5dddf4a14512d0a3f5d804d3"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.829136 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal/instance:
uuid: "22d428d011a842c693e4d23d7f4addcf"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "e24f8036fc4b4be12fd616a3311e3a2b"
server_key_iv: "c0a34ddc5dddf4a14512d0a3f5d804d3"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.834021 15782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:07:38.837364 15882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:38.838564 15782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:38.838739 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
uuid: "22d428d011a842c693e4d23d7f4addcf"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "e24f8036fc4b4be12fd616a3311e3a2b"
server_key_iv: "c0a34ddc5dddf4a14512d0a3f5d804d3"
server_key_version: "encryptionkey@0"
I20260925 05:07:38.838929 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:38.853457 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:38.857256 15782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:38.857517 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:38.876477 15782 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.60:41459
I20260925 05:07:38.876511 15933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.60:41459 every 8 connection(s)
I20260925 05:07:38.877918 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
I20260925 05:07:38.881058 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 15782
I20260925 05:07:38.882138 15934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:38.892027 15934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.60@KRBTEST.COM
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.60@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:38.904430 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.883096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.8.215.60:41459)
0925 05:07:38.883854 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.883869 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.883920 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:07:38.885751 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.885779 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.886310 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.886771 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.888163 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.890270 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.891462 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.892561 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.892784 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.897664 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.897671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.897681 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.897715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.900600 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.901285 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.901298 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.901304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.901476 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.901865 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.901870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.901873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.902330 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.902640 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.903088 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.903450 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":320,"threads_started":1}
I20260925 05:07:38.909109 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.895060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.8.215.62:36215)
0925 05:07:38.895295 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.895305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.895862 (+ 557us) server_negotiation.cc:407] Connection header received
0925 05:07:38.896080 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.896086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.896159 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.896275 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.897551 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.898283 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.899678 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.900078 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.900203 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.903275 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.903282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.903287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.903328 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.905887 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.906753 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.906759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.906761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.906963 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.907370 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.907374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.907376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.907765 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.907927 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.908544 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.908772 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:07:38.909157 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.894865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:41836)
0925 05:07:38.895701 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.895740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.895950 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.896351 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.896358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.897116 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.897392 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.897401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.898427 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.898432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.899527 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.899537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.900267 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.900278 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.900443 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.901078 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.901101 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.903102 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.906006 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.906018 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.906024 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.906627 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.907094 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.907098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.907101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.907255 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.907969 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.907984 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.908181 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.908430 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.908756 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":424,"threads_started":1}
I20260925 05:07:38.910307 15934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.60@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
I20260925 05:07:38.922341 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.911008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37616)
0925 05:07:38.911169 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.911183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.911254 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.911917 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.911920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.912189 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.912320 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.912329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.913631 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.913635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.914529 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.914539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.915208 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.915212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.915346 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.915902 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.915918 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.917874 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.920440 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.920443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.920446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.920876 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.921200 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.921203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.921205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.921323 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.921733 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.921736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.921841 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.921941 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.922129 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 05:07:38.922468 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.911057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.8.215.61:43439)
0925 05:07:38.911681 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.911685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.911700 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:07:38.911745 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.911749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.911800 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.911882 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.912475 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.913485 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.914647 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.915073 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.915163 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.918031 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.918035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.918038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.918065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.920329 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.920982 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.920986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.920988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.921110 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.921406 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.921409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.921411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.921642 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.921752 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.922023 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.922166 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":393,"threads_started":1}
I20260925 05:07:38.923491 15934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:38 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312858, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.60@KRBTEST.COM for oryx/127.8.215.60@KRBTEST.COM
I20260925 05:07:38.934772 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.924232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.0.0.1:58870)
0925 05:07:38.924412 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.924429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.924548 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.924843 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.924847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.925193 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.925311 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.925318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.926224 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.926231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.927027 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.927037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.927695 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.927701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.927893 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.928424 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.928440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.930323 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.932830 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.932834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.932836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.933172 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.933470 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.933474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.933478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.933610 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.934020 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.934023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.934156 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.934258 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.934421 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:07:38.935105 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.924208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.8.215.60:41459)
0925 05:07:38.924395 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.924401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.924494 (+ 93us) server_negotiation.cc:407] Connection header received
0925 05:07:38.924648 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.924654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.924704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.924832 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.925434 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.926105 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.927162 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.927548 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.927678 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.930490 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.930494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.930497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.930522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.932649 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.933265 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.933269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.933271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.933381 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.933696 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.933699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.933701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.933928 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.934026 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.934341 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.934551 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":70}
I20260925 05:07:38.941109 15934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Bootstrap starting.
I20260925 05:07:38.945603 15934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:38.946956 15934 log.cc:824] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:38.950089 15934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: No bootstrap required, opened a new log
I20260925 05:07:38.957561 15934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } }
I20260925 05:07:38.958029 15934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:38.958169 15934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d428d011a842c693e4d23d7f4addcf, State: Initialized, Role: FOLLOWER
I20260925 05:07:38.958913 15934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } }
I20260925 05:07:38.960206 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.953956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.8.215.61:43439)
0925 05:07:38.954181 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.954190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.954511 (+ 321us) server_negotiation.cc:407] Connection header received
0925 05:07:38.954730 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.954734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.954819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.954941 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:38.955343 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.956052 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.957893 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.959170 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.959317 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.959599 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.959853 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 05:07:38.959983 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:38.959914 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } } }
I20260925 05:07:38.960322 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:38.960558 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.953780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37620)
0925 05:07:38.954397 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.954412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.954582 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.954976 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.954981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.955008 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:38.955204 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.955212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.956204 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.956214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.957732 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.957747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.959403 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.959410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.959640 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.959706 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 05:07:38.959816 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:07:38.959973 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":393,"threads_started":1}
I20260925 05:07:38.961177 15934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:38.961686 15828 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 05:07:38.962198 15856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d648d6d50cd4c9bb2089acea54b0f6f. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } }
I20260925 05:07:38.962579 15856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: LEARNER
W20260925 05:07:38.971405 15954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:07:38.971567 15954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:07:38.976130 15782 master_runner.cc:428] Detected that this master 22d428d011a842c693e4d23d7f4addcf is joining an existing cluster
I20260925 05:07:38.976228 15782 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.215.60:41459
I20260925 05:07:39.000644 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.981257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.8.215.61:43439)
0925 05:07:38.981420 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.981427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.984101 (+ 2674us) server_negotiation.cc:407] Connection header received
0925 05:07:38.984232 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.984239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.984333 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.984453 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.985913 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.986782 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.988908 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.989558 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.989651 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.992833 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.992839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.992844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.992883 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.996703 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.998047 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.998052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.998056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.998219 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.998789 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.998795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.998799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.999167 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.999305 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.000010 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.000232 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:07:39.000634 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.980306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.8.215.62:36215)
0925 05:07:38.980525 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.980533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.981303 (+ 770us) server_negotiation.cc:407] Connection header received
0925 05:07:38.981598 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.981605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.981685 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.981754 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.985317 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.986518 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.988919 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.989557 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.989651 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.992838 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.992845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.992849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.992883 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.996682 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.998034 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.998040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.998043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.998219 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:38.998789 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:38.998795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:38.998799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:38.999167 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:38.999305 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:38.999991 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.000223 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:07:39.001935 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.980460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:41842)
0925 05:07:38.981198 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.981219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.981422 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.981967 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.981972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.982843 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.984421 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.984433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.986763 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.986774 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.988718 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.988756 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.989952 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.989960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.990272 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.991140 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.991174 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.992602 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.996864 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.996868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.996875 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.997873 (+ 998us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.998399 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.998404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.998408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.998647 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.999459 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.999463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.999695 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.000936 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.001291 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":274,"spinlock_wait_cycles":8320,"thread_start_us":1388,"threads_started":1}
I20260925 05:07:39.001982 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.981578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37632)
0925 05:07:38.983972 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.983992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.984120 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.984665 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.984670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.985525 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.985779 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.985787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.986947 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.986951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.988741 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.988756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.989956 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.989961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.990269 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.991214 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.991231 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.992605 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.996860 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.996865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.996869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.997914 (+ 1045us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.998388 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.998395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.998399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:38.998647 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:38.999392 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:38.999397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:38.999677 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.000936 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.001489 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2267,"mutex_wait_us":281,"spinlock_wait_cycles":64768,"thread_start_us":1901,"threads_started":1}
I20260925 05:07:39.002889 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.981212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.8.215.60:41459)
0925 05:07:38.981394 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:38.981399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:38.985087 (+ 3688us) server_negotiation.cc:407] Connection header received
0925 05:07:38.985233 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:38.985238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:38.985305 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:38.985392 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:38.988588 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.989693 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.991683 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.992412 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.992518 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.995327 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:38.995333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:38.995338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:38.995370 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:38.999401 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.000201 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.000206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.000209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.000353 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.001264 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.001272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.001277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.001660 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.001855 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.002445 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.002655 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:39.003859 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:38.981730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.0.0.1:58876)
0925 05:07:38.984963 (+ 3233us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:38.984985 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:38.985117 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:38.987413 (+ 2296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:38.987417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:38.988058 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:38.988427 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.988438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.989879 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.989884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.991534 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:38.991548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:38.992865 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:38.992873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:38.993036 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:38.994048 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:38.994069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:38.995150 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:38.999546 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:38.999551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:38.999555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.000085 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.000510 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.000515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.000519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.000687 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.001942 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:39.001949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:39.002140 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.003057 (+ 917us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.003358 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3089,"thread_start_us":2895,"threads_started":1}
I20260925 05:07:39.010356 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15281
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312856, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.62@KRBTEST.COM for oryx/127.8.215.60@KRBTEST.COM
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.1@KRBTEST.COM
I20260925 05:07:39.024128 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.007952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.0.0.1:58890)
0925 05:07:39.008140 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.008166 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:39.008267 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:39.008787 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:39.008798 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:39.009296 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:39.009589 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.009603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.011140 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.011146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.012569 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.012584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.013671 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.013679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.013911 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.014634 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:39.014661 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:39.017298 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:39.021370 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.021378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.021383 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.021779 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.022211 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.022217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.022221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.022424 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.023016 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:39.023021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:39.023216 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.023367 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.023655 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:07:39.024128 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.007969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.8.215.60:41459)
0925 05:07:39.008139 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.008145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.008240 (+ 95us) server_negotiation.cc:407] Connection header received
0925 05:07:39.008512 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.008518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.008611 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.008711 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.009789 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.010973 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.012789 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.013489 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.013621 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.017518 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.017524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.017527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.017564 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.021214 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.021935 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.021941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.021945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.022075 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.022572 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.022579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.022583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.022889 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.023048 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.023481 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.023710 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:07:39.026280 15219 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.215.60:41459: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "22d428d011a842c693e4d23d7f4addcf"
member_type: NON_VOTER
last_known_addr {
host: "127.8.215.60"
port: 41459
}
attrs {
promote: true
}
}
dest_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f"
cas_config_opid_index: 5
I20260925 05:07:39.027401 15219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } }
I20260925 05:07:39.030493 15909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:39.033450 15828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 LEARNER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:07:39.033946 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.011519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.8.215.1:39177)
0925 05:07:39.012158 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.012169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.012201 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:07:39.012286 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.012295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.012582 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.012798 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.014426 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.016166 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.018188 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.019226 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.019380 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.025448 (+ 6068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.025467 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.025487 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.025543 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.029390 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.030409 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.030419 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.030423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.030597 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.031096 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.031101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.031104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.031625 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.031944 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.033209 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.033499 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":611,"threads_started":1}
I20260925 05:07:39.035295 15404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:39.035137 15941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 05:07:39.037106 15964 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 22d428d011a842c693e4d23d7f4addcf (127.8.215.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.040078 15964 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.215.60:41459
I20260925 05:07:39.040304 15964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.040591 15964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:39.041767 15828 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 22d428d011a842c693e4d23d7f4addcf (127.8.215.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.048002 15856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: NON_VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: true } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.048314 15856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: LEARNER
I20260925 05:07:39.050664 15941 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: attempting to promote NON_VOTER 2aafd2c3c69c4d0897dc79edf79fdd3b to VOTER
I20260925 05:07:39.051463 15941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } }
I20260925 05:07:39.057578 15909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:07:39.058367 15782 master.cc:537] Master@127.8.215.60:41459 shutting down...
I20260925 05:07:39.060281 15908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 05:07:39.061877 15828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 LEARNER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 05:07:39.062078 15969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:39.062821 15941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 05:07:39.066429 15828 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2aafd2c3c69c4d0897dc79edf79fdd3b (127.8.215.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.072289 15856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.072537 15856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:39.065500 15969 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2aafd2c3c69c4d0897dc79edf79fdd3b (127.8.215.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.074427 15782 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:39.074821 15782 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:39.075026 15782 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: stopping tablet replica
I20260925 05:07:39.080385 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.080786 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:39.081395 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.081730 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:39.082338 15782 master.cc:552] Master@127.8.215.60:41459 shutdown complete.
I20260925 05:07:39.082434 15782 master_runner.cc:305] Clearing existing system tablet
I20260925 05:07:39.082826 15782 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:39.082947 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:39.086933 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.086385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.0.0.1:58906)
0925 05:07:39.086563 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.086746 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:41459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 05:07:39.087241 15207 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f -> Peer 22d428d011a842c693e4d23d7f4addcf (127.8.215.60:41459): Couldn't send request to peer 22d428d011a842c693e4d23d7f4addcf. Status: Network error: Client connection negotiation failed: client connection to 127.8.215.60:41459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:07:39.088984 15782 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:07:39.090691 15972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:39.091049 15782 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:39.091351 15782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:07:39.091475 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
uuid: "22d428d011a842c693e4d23d7f4addcf"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "e24f8036fc4b4be12fd616a3311e3a2b"
server_key_iv: "c0a34ddc5dddf4a14512d0a3f5d804d3"
server_key_version: "encryptionkey@0"
I20260925 05:07:39.095606 15782 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:39.100492 15782 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:07:39.100677 15782 log.cc:1196] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 05:07:39.101161 15782 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Deleting consensus metadata
I20260925 05:07:39.101480 15782 master_runner.cc:315] Copying system tablet from 127.8.215.62:36215
I20260925 05:07:39.104089 15782 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.215.62:36215
I20260925 05:07:39.121569 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.104925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.8.215.62:36215)
0925 05:07:39.105231 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.105241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.109305 (+ 4064us) server_negotiation.cc:407] Connection header received
0925 05:07:39.109552 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.109563 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.109631 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.109747 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.110664 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.111421 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.112790 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.113257 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.113410 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.115400 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.115406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.115410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.115443 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.118621 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.119354 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.119361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.119365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.119508 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.119946 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.119954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.119960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.120265 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.120457 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.120873 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.121117 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 05:07:39.121613 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.104913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:41850)
0925 05:07:39.109161 (+ 4248us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.109186 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:39.109396 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:39.109772 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:39.109777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:39.110229 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:39.110471 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.110481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.111585 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.111591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.112579 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.112592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.113435 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.113441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.113659 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.114390 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:39.114409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:39.115171 (+ 762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:39.118789 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.118797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.118802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.119210 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.119640 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.119646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.119650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.119826 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.120397 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:39.120402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:39.120595 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.120800 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.121122 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4105,"thread_start_us":4001,"threads_started":1}
I20260925 05:07:39.122561 15240 tablet_copy_service.cc:140] P 6d648d6d50cd4c9bb2089acea54b0f6f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 22d428d011a842c693e4d23d7f4addcf ({username='oryx', principal='oryx/127.8.215.60@KRBTEST.COM'} at 127.0.0.1:41850)
I20260925 05:07:39.122745 15240 tablet_copy_service.cc:161] P 6d648d6d50cd4c9bb2089acea54b0f6f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 22d428d011a842c693e4d23d7f4addcf at {username='oryx', principal='oryx/127.8.215.60@KRBTEST.COM'} at 127.0.0.1:41850: session id = 22d428d011a842c693e4d23d7f4addcf-00000000000000000000000000000000
I20260925 05:07:39.127077 15240 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:07:39.128958 15782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:07:39.135493 15782 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: tablet copy: Starting download of 0 data blocks...
I20260925 05:07:39.135771 15782 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: tablet copy: Starting download of 1 WAL segments...
I20260925 05:07:39.138082 15782 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:07:39.142318 15240 tablet_copy_service.cc:342] P 6d648d6d50cd4c9bb2089acea54b0f6f: Request end of tablet copy session 22d428d011a842c693e4d23d7f4addcf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.215.60@KRBTEST.COM'} at 127.0.0.1:41850
I20260925 05:07:39.142462 15240 tablet_copy_service.cc:433] P 6d648d6d50cd4c9bb2089acea54b0f6f: ending tablet copy session 22d428d011a842c693e4d23d7f4addcf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 22d428d011a842c693e4d23d7f4addcf
I20260925 05:07:39.153788 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:39.156639 15979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:39.158062 15980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:39.169353 15982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:39.169929 15782 server_base.cc:1034] running on GCE node
I20260925 05:07:39.170269 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:39.170886 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:39.172097 15782 hybrid_clock.cc:648] HybridClock initialized: now 1790312859172054 us; error 24 us; skew 500 ppm
I20260925 05:07:39.172328 15782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312859, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:39.174422 15782 init.cc:377] Logged in from keytab as oryx/127.8.215.60@KRBTEST.COM (short username oryx)
I20260925 05:07:39.175673 15782 webserver.cc:460] Webserver started at http://127.8.215.60:40543/ using document root <none> and password file <none>
I20260925 05:07:39.176255 15782 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:39.176362 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:39.179504 15782 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:07:39.181416 15989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:39.181811 15782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:39.181955 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
uuid: "22d428d011a842c693e4d23d7f4addcf"
format_stamp: "Formatted at 2026-09-25 05:07:38 on dist-test-slave-wpdm"
server_key: "e24f8036fc4b4be12fd616a3311e3a2b"
server_key_iv: "c0a34ddc5dddf4a14512d0a3f5d804d3"
server_key_version: "encryptionkey@0"
I20260925 05:07:39.182165 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:39.203953 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:39.208423 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:39.218272 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15281
I20260925 05:07:39.249505 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:39177
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:36215
--webserver_port=44339
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:39.252053 15782 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.60:41459
I20260925 05:07:39.252189 16040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.60:41459 every 8 connection(s)
I20260925 05:07:39.253235 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
I20260925 05:07:39.258693 16041 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:07:39.262782 16041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Bootstrap starting.
I20260925 05:07:39.282207 16041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:07:39.282804 16041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: Bootstrap complete.
I20260925 05:07:39.283955 16041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } }
I20260925 05:07:39.284241 16041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22d428d011a842c693e4d23d7f4addcf, State: Initialized, Role: LEARNER
I20260925 05:07:39.284495 16041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } }
I20260925 05:07:39.285396 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.285741 16041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:39.286006 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: This master's current role is: LEARNER
I20260925 05:07:39.297281 16056 catalog_manager.cc:1259] Loaded cluster ID: dfbfdca724914b0baa3b67d52b054438
I20260925 05:07:39.297520 16056 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: loading cluster ID for follower catalog manager: success
I20260925 05:07:39.301254 16056 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: acquiring CA information for follower catalog manager: success
I20260925 05:07:39.303962 16056 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:07:39.600525 16043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:39.600914 16043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:39.601089 16043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:39.608505 16043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:39.608664 16043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:39.608862 16043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:39.618836 16043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:39177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=44339
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:39.620249 16043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:39.621595 16043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:39.634580 16061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:39.634529 16064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:39.634773 16062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:39.635485 16043 server_base.cc:1034] running on GCE node
I20260925 05:07:39.636096 16043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:39.637073 16043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:39.638265 16043 hybrid_clock.cc:648] HybridClock initialized: now 1790312859638229 us; error 39 us; skew 500 ppm
I20260925 05:07:39.638569 16043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312859, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:39.641515 16043 init.cc:377] Logged in from keytab as oryx/127.8.215.1@KRBTEST.COM (short username oryx)
I20260925 05:07:39.642902 16043 webserver.cc:460] Webserver started at http://127.8.215.1:44339/ using document root <none> and password file <none>
I20260925 05:07:39.643551 16043 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:39.643659 16043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:39.650029 16043 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:07:39.653380 16071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:39.655370 16043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260925 05:07:39.655558 16043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "d2cf57f6810e4f278a02eeae532b7258"
format_stamp: "Formatted at 2026-09-25 05:07:36 on dist-test-slave-wpdm"
server_key: "27343f3439e7612c80b3147f0ce98631"
server_key_iv: "5724343ccab6479470474ee494a21fe2"
server_key_version: "encryptionkey@0"
I20260925 05:07:39.657034 16043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:39.660138 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.653129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.0.0.1:58914)
0925 05:07:39.653594 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.653615 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:39.653761 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:39.654365 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:39.654372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:39.654405 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:39.654613 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.654655 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.656599 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.656603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.657929 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.657942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.659599 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.659603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.659732 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.659757 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.659821 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.659915 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
I20260925 05:07:39.660372 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.653293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.8.215.60:41459)
0925 05:07:39.653931 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.653935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.653987 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:07:39.654066 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.654071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.654185 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.654336 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:39.654789 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.656413 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.658097 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.659416 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.659584 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.659802 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.659859 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.660028 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":375,"threads_started":1}
I20260925 05:07:39.661442 16016 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260925 05:07:39.662118 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d648d6d50cd4c9bb2089acea54b0f6f. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:39.662487 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: This master's current role is: LEARNER
I20260925 05:07:39.692562 16043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:39.697280 16043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:39.697597 16043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:39.698729 16043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:39.700148 16043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:39.700280 16043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:39.700428 16043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:39.700536 16043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:39.740446 16043 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:39177
I20260925 05:07:39.740549 16185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:39177 every 8 connection(s)
I20260925 05:07:39.742097 16043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:39.747120 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 16043
I20260925 05:07:39.747527 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15415
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.2@KRBTEST.COM
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312859, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312859, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
Sep 25 05:07:39 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312859, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.60@KRBTEST.COM
I20260925 05:07:39.795369 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.749399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.8.215.2:34959)
0925 05:07:39.771213 (+ 21814us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.771225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.773148 (+ 1923us) server_negotiation.cc:407] Connection header received
0925 05:07:39.773299 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.773308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.773543 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.773744 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.775703 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.777525 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.779542 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.780576 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.780691 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.787188 (+ 6497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.787203 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.787222 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.787269 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.791339 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.792216 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.792226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.792230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.792389 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.792909 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.792950 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.792952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.793380 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.793668 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.794713 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.794985 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21581,"thread_start_us":22328,"threads_started":1}
I20260925 05:07:39.796844 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.745427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:60331 (local address 127.8.215.61:43439)
0925 05:07:39.774408 (+ 28981us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.774425 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.774453 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:07:39.775165 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.775170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.775259 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.776051 (+ 792us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.778598 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.779661 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.781808 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.782580 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.782710 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.788865 (+ 6155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.788912 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.788917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.788958 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.792775 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.793668 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.793674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.793677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.793822 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.794487 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.794493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.794498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.794815 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.794947 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.796262 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.796484 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28851,"thread_start_us":29016,"threads_started":1}
I20260925 05:07:39.797322 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.747072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:44333 (local address 127.8.215.62:36215)
0925 05:07:39.769579 (+ 22507us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.769590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.772311 (+ 2721us) server_negotiation.cc:407] Connection header received
0925 05:07:39.775092 (+ 2781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.775100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.775189 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.775277 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.778252 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.779336 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.782496 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.783110 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.783258 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.789778 (+ 6520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.789787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.789795 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.789842 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.793630 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.794698 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.794704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.794708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.794883 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.795417 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.795425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.795431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.795814 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.795960 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.796583 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.796847 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22341,"thread_start_us":22233,"threads_started":1}
I20260925 05:07:39.797394 15538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:39.799525 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.746254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:44333)
0925 05:07:39.772193 (+ 25939us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.772210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:39.774770 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:39.775677 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:39.775689 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:39.776843 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:39.778079 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.778101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.779504 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.779510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.781627 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.781647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.783390 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.783399 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.783875 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.784605 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:39.784637 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:39.789538 (+ 4901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:39.793849 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.793857 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.793867 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.794554 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.795076 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.795085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.795090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.795271 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.796047 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:39.796053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:39.796247 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.797512 (+ 1265us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.797926 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25794,"mutex_wait_us":3507,"spinlock_wait_cycles":343296,"thread_start_us":23140,"threads_started":1}
I20260925 05:07:39.801224 16187 heartbeater.cc:347] Connected to a master server at 127.8.215.62:36215
I20260925 05:07:39.801596 16187 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:39.802219 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.745875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:36697 (local address 127.8.215.60:41459)
0925 05:07:39.746155 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:39.746164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:39.770326 (+ 24162us) server_negotiation.cc:407] Connection header received
0925 05:07:39.775080 (+ 4754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:39.775085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:39.775168 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:39.775241 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:39.780876 (+ 5635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.782365 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.786653 (+ 4288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.787390 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.787549 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.793013 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:39.793021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:39.793027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:39.793072 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:39.797492 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.799129 (+ 1637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.799138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.799145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.799333 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:39.800279 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:39.800287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:39.800293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:39.800719 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:39.800923 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:39.801572 (+ 649us) server_negotiation.cc:299] Negotiation successful
0925 05:07:39.801869 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 05:07:39.802528 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.746254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.8.215.1:36697)
0925 05:07:39.770081 (+ 23827us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.770151 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:39.774744 (+ 4593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:39.775675 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:39.775687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:39.776835 (+ 1148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:39.778081 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.778104 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.783518 (+ 5414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.783524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.786434 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.786448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.787669 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.787677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.787850 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.788740 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:39.788770 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:39.792763 (+ 3993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:39.798154 (+ 5391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.798158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.798162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.798876 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.799524 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.799531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.799535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.800095 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.801000 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:39.801008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:39.801189 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.801427 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.801818 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21826,"mutex_wait_us":130,"spinlock_wait_cycles":61696,"thread_start_us":25326,"threads_started":1}
I20260925 05:07:39.802688 16187 heartbeater.cc:511] Master 127.8.215.62:36215 requested a full tablet report, sending...
I20260925 05:07:39.803785 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:39.746328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.8.215.1:60331)
0925 05:07:39.772080 (+ 25752us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:39.772099 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:39.774744 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:39.775675 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:39.775687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:39.776835 (+ 1148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:39.778367 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.778379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.779810 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.779815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.781607 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:39.781620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:39.783074 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:39.783083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:39.783506 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:39.784425 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:39.784461 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:39.788547 (+ 4086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:39.792917 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.792926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.792958 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.793535 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.793962 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:39.793967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:39.793972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:39.794341 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:39.795025 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:39.795033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:39.795554 (+ 521us) client_negotiation.cc:769] Sending connection context
0925 05:07:39.795896 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 05:07:39.803405 (+ 7509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25605,"mutex_wait_us":1739,"spinlock_wait_cycles":10368,"thread_start_us":18365,"threads_started":1}
I20260925 05:07:39.804013 16186 heartbeater.cc:347] Connected to a master server at 127.8.215.60:41459
I20260925 05:07:39.804126 16186 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:39.804526 16186 heartbeater.cc:511] Master 127.8.215.60:41459 requested a full tablet report, sending...
I20260925 05:07:39.804607 15219 ts_manager.cc:194] Re-registered known tserver with Master: d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177)
I20260925 05:07:39.805627 16188 heartbeater.cc:347] Connected to a master server at 127.8.215.61:43439
I20260925 05:07:39.805755 16188 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:39.806022 15219 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:44333
I20260925 05:07:39.806180 16188 heartbeater.cc:511] Master 127.8.215.61:43439 requested a full tablet report, sending...
I20260925 05:07:39.806351 16006 ts_manager.cc:194] Registered new tserver with Master: d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177)
I20260925 05:07:39.807749 15818 ts_manager.cc:194] Registered new tserver with Master: d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177)
I20260925 05:07:39.889979 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15415
I20260925 05:07:39.907570 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:34959
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:36215
--webserver_port=39883
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:40.033294 15969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: NON_VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: true } } }
I20260925 05:07:40.033643 15969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:40.035316 16200 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: attempting to promote NON_VOTER 22d428d011a842c693e4d23d7f4addcf to VOTER
I20260925 05:07:40.035876 16200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } }
I20260925 05:07:40.037772 15828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 05:07:40.038205 16016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 LEARNER]: Refusing update from remote peer 6d648d6d50cd4c9bb2089acea54b0f6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 05:07:40.038403 16199 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 05:07:40.038812 15969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 05:07:40.041320 16199 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 22d428d011a842c693e4d23d7f4addcf (127.8.215.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.041879 15828 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 22d428d011a842c693e4d23d7f4addcf (127.8.215.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.045534 15969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.045949 15969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:40.046312 15969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.046578 15969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:40.046944 15969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.046528 16016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 22d428d011a842c693e4d23d7f4addcf (127.8.215.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.047227 15969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:40.051314 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.051563 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:40.051781 16203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:40.052206 16203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:07:40.237524 16198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:40.237807 16198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:40.237900 16198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:40.245110 16198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:40.245218 16198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:40.245332 16198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:40.255080 16198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:34959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=39883
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:40.256333 16198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:40.257853 16198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:40.271376 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:40.271334 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:40.271402 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:40.274299 16198 server_base.cc:1034] running on GCE node
I20260925 05:07:40.274921 16198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:40.275848 16198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:40.277070 16198 hybrid_clock.cc:648] HybridClock initialized: now 1790312860277035 us; error 54 us; skew 500 ppm
I20260925 05:07:40.277386 16198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:40.280560 16198 init.cc:377] Logged in from keytab as oryx/127.8.215.2@KRBTEST.COM (short username oryx)
I20260925 05:07:40.282198 16198 webserver.cc:460] Webserver started at http://127.8.215.2:39883/ using document root <none> and password file <none>
I20260925 05:07:40.282998 16198 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:40.283155 16198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:40.288540 16198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:40.291261 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:40.292546 16198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:40.292748 16198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "5cc7a29b523b433b8056553dd4a77ae0"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "b51d713a72947b2ef50034a6e7bd697b"
server_key_iv: "13151834231c4b1854e314239c99284c"
server_key_version: "encryptionkey@0"
I20260925 05:07:40.293776 16198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:40.315470 16198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:40.319819 16198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:40.320091 16198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:40.321079 16198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:40.322418 16198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:40.322546 16198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:40.322679 16198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:40.322767 16198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:40.355872 16198 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:34959
I20260925 05:07:40.355959 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:34959 every 8 connection(s)
I20260925 05:07:40.357405 16198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:40.359681 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 16198
I20260925 05:07:40.359956 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15548
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.60@KRBTEST.COM
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.3@KRBTEST.COM
I20260925 05:07:40.397114 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.360343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:53817 (local address 127.8.215.60:41459)
0925 05:07:40.375614 (+ 15271us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.375620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.376281 (+ 661us) server_negotiation.cc:407] Connection header received
0925 05:07:40.377995 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.378003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.378094 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.378222 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.380102 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.381246 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.383407 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.384062 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.384180 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.389611 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:40.389616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:40.389619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:40.389649 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:40.393256 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.394284 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.394289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.394293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.394442 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.395016 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.395022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.395028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.395372 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:40.395484 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:40.396497 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0925 05:07:40.396719 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15142,"thread_start_us":15284,"threads_started":1}
I20260925 05:07:40.398715 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.362820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.8.215.2:53817)
0925 05:07:40.376165 (+ 13345us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:40.376195 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:40.377178 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:40.378454 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:40.378459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:40.379708 (+ 1249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:40.379945 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.379955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.381554 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.381563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.383279 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.383290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.384288 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.384293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.384476 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.385351 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:40.385397 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:40.389328 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:40.393468 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.393479 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.393498 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.394143 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:40.394619 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.394623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.394627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.394882 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:40.395589 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:40.395598 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:40.396039 (+ 441us) client_negotiation.cc:769] Sending connection context
0925 05:07:40.397370 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0925 05:07:40.397617 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13205,"spinlock_wait_cycles":13184,"thread_start_us":13596,"threads_started":1}
I20260925 05:07:40.398782 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.362356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.8.215.3:39793)
0925 05:07:40.375046 (+ 12690us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.375058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.375095 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:07:40.375175 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.375187 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.375488 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.375678 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.378758 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.380395 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.382308 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.384062 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.384180 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.390598 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:40.390607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:40.390620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:40.390652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:40.394681 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.395585 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.395596 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.395599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.395754 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.396184 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.396188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.396190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.396548 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:40.396857 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:40.398108 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0925 05:07:40.398372 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12507,"thread_start_us":12349,"threads_started":1}
I20260925 05:07:40.399722 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.361740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:48525 (local address 127.8.215.61:43439)
0925 05:07:40.373419 (+ 11679us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.373425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.373449 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:07:40.375211 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.375216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.375319 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.375427 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.379022 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.380258 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.383083 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.383618 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.383784 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.390500 (+ 6716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:40.390506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:40.390511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:40.390565 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:40.394525 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.395422 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.395428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.395431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.395572 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.396006 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.396011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.396014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.396350 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:40.396496 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:40.399147 (+ 2651us) server_negotiation.cc:299] Negotiation successful
0925 05:07:40.399379 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11192,"thread_start_us":11532,"threads_started":1}
I20260925 05:07:40.400023 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.361082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.8.215.2:48525)
0925 05:07:40.370645 (+ 9563us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:40.370709 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:40.374857 (+ 4148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:40.375891 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:40.375907 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:40.376866 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:40.378704 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.378729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.380449 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.380454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.382908 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.382921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.384031 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.384037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.384175 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.385003 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:40.385027 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:40.389959 (+ 4932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:40.394736 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.394743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.394750 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.395280 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:40.395702 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.395706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.395709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.395868 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:40.396480 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:40.396485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:40.396630 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 05:07:40.396800 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 05:07:40.397359 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9232,"mutex_wait_us":96,"spinlock_wait_cycles":607616,"thread_start_us":8942,"threads_started":1}
I20260925 05:07:40.400218 15671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:40.400662 16331 heartbeater.cc:347] Connected to a master server at 127.8.215.60:41459
I20260925 05:07:40.401093 16331 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:40.401196 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.361083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:39291)
0925 05:07:40.376916 (+ 15833us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:40.376935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:40.377341 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:40.378053 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:40.378061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:40.378818 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:40.379112 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.379122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.380679 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.380684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.382913 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.382928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.383878 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.383886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.384169 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.385393 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:40.385433 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:40.390369 (+ 4936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:40.395424 (+ 5055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.395429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.395434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.396016 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:40.398692 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.398698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.398705 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.398875 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:40.400194 (+ 1319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:40.400199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:40.400321 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 05:07:40.400481 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 05:07:40.400809 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15682,"mutex_wait_us":2143,"spinlock_wait_cycles":252032,"thread_start_us":12976,"threads_started":1}
I20260925 05:07:40.402172 16331 heartbeater.cc:511] Master 127.8.215.60:41459 requested a full tablet report, sending...
I20260925 05:07:40.404157 16006 ts_manager.cc:194] Registered new tserver with Master: 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959)
I20260925 05:07:40.404451 16333 heartbeater.cc:347] Connected to a master server at 127.8.215.61:43439
I20260925 05:07:40.404585 16333 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:40.405066 16333 heartbeater.cc:511] Master 127.8.215.61:43439 requested a full tablet report, sending...
I20260925 05:07:40.406008 15818 ts_manager.cc:194] Registered new tserver with Master: 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959)
I20260925 05:07:40.406037 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.360279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:39291 (local address 127.8.215.62:36215)
0925 05:07:40.376899 (+ 16620us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.376905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.377043 (+ 138us) server_negotiation.cc:407] Connection header received
0925 05:07:40.377496 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.377502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.377588 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.377682 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.379270 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.380319 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.383043 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.383624 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.383746 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.391328 (+ 7582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:40.391333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:40.391336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:40.391368 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:40.394992 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.396570 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.396575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.396578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.396761 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.399615 (+ 2854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.399619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.399621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:40.399938 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:40.404351 (+ 4413us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:40.404593 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 05:07:40.405520 (+ 927us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16507,"thread_start_us":16835,"threads_started":1}
I20260925 05:07:40.407070 16332 heartbeater.cc:347] Connected to a master server at 127.8.215.62:36215
I20260925 05:07:40.407255 16332 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:40.407833 16332 heartbeater.cc:511] Master 127.8.215.62:36215 requested a full tablet report, sending...
I20260925 05:07:40.409060 15219 ts_manager.cc:194] Re-registered known tserver with Master: 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959)
I20260925 05:07:40.410193 15219 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:39291
I20260925 05:07:40.490274 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15548
I20260925 05:07:40.507165 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:39793
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:36215
--webserver_port=38865
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:40.809685 16187 heartbeater.cc:503] Master 127.8.215.62:36215 was elected leader, sending a full tablet report...
W20260925 05:07:40.851588 16344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:40.851845 16344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:40.851936 16344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:40.858956 16344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:40.859068 16344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:40.859191 16344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:40.869445 16344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:39793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=38865
--tserver_master_addrs=127.8.215.62:36215,127.8.215.61:43439,127.8.215.60:41459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:40.870648 16344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:40.871868 16344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:40.885011 16349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:40.885037 16352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:40.885119 16350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:40.885846 16344 server_base.cc:1034] running on GCE node
I20260925 05:07:40.886559 16344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:40.887624 16344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:40.888829 16344 hybrid_clock.cc:648] HybridClock initialized: now 1790312860888761 us; error 51 us; skew 500 ppm
I20260925 05:07:40.889178 16344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:40.892254 16344 init.cc:377] Logged in from keytab as oryx/127.8.215.3@KRBTEST.COM (short username oryx)
I20260925 05:07:40.893724 16344 webserver.cc:460] Webserver started at http://127.8.215.3:38865/ using document root <none> and password file <none>
I20260925 05:07:40.894443 16344 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:40.894613 16344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:40.900028 16344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 05:07:40.902763 16359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:40.904034 16344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:40.904155 16344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "3ce603504d2b4ea1902c513b97272c52"
format_stamp: "Formatted at 2026-09-25 05:07:37 on dist-test-slave-wpdm"
server_key: "a5af1f2aaa958cbf429fd59bff26822b"
server_key_iv: "df8b240d4f66d5dfcea04156c8fb8122"
server_key_version: "encryptionkey@0"
I20260925 05:07:40.905129 16344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:40.922711 16344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:40.926578 16344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:40.926795 16344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:40.927667 16344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:40.928965 16344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:40.929059 16344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:40.929138 16344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:40.929193 16344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:40.965111 16344 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39793
I20260925 05:07:40.965137 16471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39793 every 8 connection(s)
I20260925 05:07:40.966568 16344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:40.967618 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 16344
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
Sep 25 05:07:40 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.60@KRBTEST.COM
I20260925 05:07:41.004966 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.971399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:43327)
0925 05:07:40.978593 (+ 7194us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:40.978614 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:40.978840 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:40.979785 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:40.979803 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:40.982359 (+ 2556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:40.984077 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.984096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.985306 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.985312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.987512 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.987527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.988842 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.988851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.989133 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.989995 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:40.990027 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:40.993901 (+ 3874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:40.998783 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:40.998795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:40.998814 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:40.999508 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:41.000352 (+ 844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:41.000361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:41.000367 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:41.000643 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:41.001590 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:41.001605 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:41.002098 (+ 493us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.002478 (+ 380us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.003187 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7047,"mutex_wait_us":290,"thread_start_us":5631,"threads_started":1}
I20260925 05:07:41.006347 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.969419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:43327 (local address 127.8.215.62:36215)
0925 05:07:40.977692 (+ 8273us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.977711 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.978721 (+ 1010us) server_negotiation.cc:407] Connection header received
0925 05:07:40.979007 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.979015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.979130 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.979219 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.984235 (+ 5016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.985126 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.987855 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.988593 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.988763 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.994206 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:40.994215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:40.994223 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:40.994274 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:40.998605 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:40.999938 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:40.999947 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:40.999953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.000170 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.000824 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.000832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.000838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.001305 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.001491 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.005715 (+ 4224us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.005975 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8106,"thread_start_us":7990,"threads_started":1}
I20260925 05:07:41.007519 16473 heartbeater.cc:347] Connected to a master server at 127.8.215.62:36215
I20260925 05:07:41.007860 16473 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:41.009256 16473 heartbeater.cc:511] Master 127.8.215.62:36215 requested a full tablet report, sending...
I20260925 05:07:41.011044 15219 ts_manager.cc:194] Re-registered known tserver with Master: 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
Sep 25 05:07:41 dist-test-slave-wpdm krb5kdc[15175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312860, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.61@KRBTEST.COM
I20260925 05:07:41.012217 15219 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:43327
I20260925 05:07:41.019763 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.970225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.8.215.3:60099)
0925 05:07:40.977329 (+ 7104us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:40.977352 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:40.977519 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:40.981538 (+ 4019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:40.981542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:40.982103 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:40.984105 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.984114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.986256 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.986260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.006148 (+ 19888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.006162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.007787 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.007797 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.008038 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.008758 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:41.008787 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:41.012115 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:41.016861 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:41.016869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:41.016875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:41.017363 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:41.017832 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:41.017838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:41.017842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:41.018030 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:41.018680 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:41.018686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:41.018817 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.019002 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.019294 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6912,"mutex_wait_us":706,"thread_start_us":7480,"threads_started":1}
I20260925 05:07:41.020808 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.970570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:60099 (local address 127.8.215.61:43439)
0925 05:07:40.981162 (+ 10592us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.981171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.981196 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:07:40.981266 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.981274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.981382 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.981508 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.984918 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.986044 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.006322 (+ 20278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.007335 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.007508 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.012349 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.012359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.012367 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.012554 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.016667 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.017513 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.017520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.017523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.017700 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.018153 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.018159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.018162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.018464 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.018617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.020198 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.020422 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10414,"thread_start_us":10293,"threads_started":1}
I20260925 05:07:41.021632 16474 heartbeater.cc:347] Connected to a master server at 127.8.215.61:43439
I20260925 05:07:41.021775 16474 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:41.022253 16474 heartbeater.cc:511] Master 127.8.215.61:43439 requested a full tablet report, sending...
I20260925 05:07:41.023559 15818 ts_manager.cc:194] Registered new tserver with Master: 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:41.027129 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.970226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.8.215.3:47955)
0925 05:07:40.971242 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:40.971305 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:40.980333 (+ 9028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:40.981068 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:40.981075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:40.981796 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:40.983071 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.983096 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.985108 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.985117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.988915 (+ 3798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:40.988929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.990252 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.990258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:40.990492 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.994953 (+ 4461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:40.994969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:40.998713 (+ 3744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:41.023941 (+ 25228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:41.023946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:41.023950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:41.024370 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:41.024788 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:41.024795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:41.024800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:41.024961 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:41.025925 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:41.025933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:41.026116 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.026251 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.026626 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":74,"thread_start_us":2953,"threads_started":1}
I20260925 05:07:41.028270 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.970953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:47955 (local address 127.8.215.60:41459)
0925 05:07:40.976790 (+ 5837us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:40.976800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:40.976854 (+ 54us) server_negotiation.cc:407] Connection header received
0925 05:07:40.980551 (+ 3697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:40.980560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:40.980668 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:40.980807 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:40.983333 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.984866 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.989083 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:40.989906 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:40.990029 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:40.998924 (+ 8895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:40.998930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:40.998939 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:40.998980 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.023801 (+ 24821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.024504 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.024507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.024509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.024618 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.025374 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.025381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.025387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.025740 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.027509 (+ 1769us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.027634 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.027863 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5651,"thread_start_us":5511,"threads_started":1}
I20260925 05:07:41.029163 16472 heartbeater.cc:347] Connected to a master server at 127.8.215.60:41459
I20260925 05:07:41.029296 16472 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:41.029647 16472 heartbeater.cc:511] Master 127.8.215.60:41459 requested a full tablet report, sending...
I20260925 05:07:41.031026 16006 ts_manager.cc:194] Registered new tserver with Master: 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:41.031591 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.989092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.8.215.61:43439)
0925 05:07:41.002154 (+ 13062us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.002162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.006447 (+ 4285us) server_negotiation.cc:407] Connection header received
0925 05:07:41.006524 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.006531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.006606 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.006702 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.013217 (+ 6515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.014633 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.017217 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.017917 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.018027 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.022773 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.022779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.022786 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.022872 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.026562 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.029039 (+ 2477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.029046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.029050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.029214 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.030029 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.030034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.030036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.030375 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.030634 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.030995 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.031202 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12917,"thread_start_us":12810,"threads_started":1}
I20260925 05:07:41.033142 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.988226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.8.215.62:36215)
0925 05:07:41.004983 (+ 16757us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.004989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.007094 (+ 2105us) server_negotiation.cc:407] Connection header received
0925 05:07:41.007145 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.007149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.007211 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.007303 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.011973 (+ 4670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.013039 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.019564 (+ 6525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.020243 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.020364 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.025285 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.025292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.025297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.025339 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.028862 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.030001 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.030009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.030013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.030158 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.031197 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.031202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.031205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.031519 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.032195 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.032527 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.032784 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16622,"thread_start_us":16511,"threads_started":1}
I20260925 05:07:41.035300 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:40.995389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.8.215.60:41459)
0925 05:07:41.014084 (+ 18695us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.014088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.014552 (+ 464us) server_negotiation.cc:407] Connection header received
0925 05:07:41.014906 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.014912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.015006 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.015243 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.017818 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.019597 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.021981 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.022802 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.022930 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.028236 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.028243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.028251 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.028302 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.032375 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.033219 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.033223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.033227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.033375 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.033785 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.033789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.033792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.034176 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.034423 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.034716 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.034954 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18587,"thread_start_us":19460,"threads_started":1}
I20260925 05:07:41.037572 15219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:41.044687 15219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:41.076159 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.067004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.8.215.3:39793)
0925 05:07:41.068159 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.068169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.068790 (+ 621us) server_negotiation.cc:407] Connection header received
0925 05:07:41.068948 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.068956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.069224 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.069439 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:41.070230 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.071205 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.073956 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.075087 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.075228 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.075453 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.075696 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.075847 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":781,"threads_started":1}
I20260925 05:07:41.076182 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.066733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39793 (local address 127.0.0.1:43158)
0925 05:07:41.068513 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:41.068534 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:41.068657 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:41.069478 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:41.069484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:41.069518 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:41.070011 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.070026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.071329 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.071337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.073755 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.073768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.075286 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.075292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.075457 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.075527 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.075645 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.075776 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1627,"thread_start_us":1472,"threads_started":1}
I20260925 05:07:41.077386 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.063885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.8.215.2:34959)
0925 05:07:41.067032 (+ 3147us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.067041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.067073 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:07:41.067214 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.067230 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.067588 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.067816 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:41.068775 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.070081 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.073931 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.075238 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.075356 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.075536 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.076981 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.077138 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3001,"thread_start_us":2920,"threads_started":1}
I20260925 05:07:41.078137 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.063672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:34959 (local address 127.0.0.1:36410)
0925 05:07:41.065703 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:41.065724 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:41.065845 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:41.068062 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:41.068069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:41.068098 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:41.068568 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.068584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.071343 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.071347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.073738 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.073753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.075668 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.075674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.076762 (+ 1088us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.076798 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.077593 (+ 795us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.077744 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1890,"mutex_wait_us":31,"thread_start_us":1766,"threads_started":1}
I20260925 05:07:41.080765 16407 tablet_service.cc:1467] Processing CreateTablet for tablet b720c75e7bc3472fabcdb34052a47f19 (DEFAULT_TABLE table=test-table [id=18e06e1f90f644f28286728f672fbbe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:41.081616 16266 tablet_service.cc:1467] Processing CreateTablet for tablet b720c75e7bc3472fabcdb34052a47f19 (DEFAULT_TABLE table=test-table [id=18e06e1f90f644f28286728f672fbbe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:41.082268 16407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b720c75e7bc3472fabcdb34052a47f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:41.082832 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b720c75e7bc3472fabcdb34052a47f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:41.082850 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.064305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:39177 (local address 127.0.0.1:53878)
0925 05:07:41.066489 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:41.066502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:41.066587 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:41.073709 (+ 7122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:41.073714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:41.073740 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:41.074157 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.074170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.075940 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.075945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.080461 (+ 4516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.080473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.081899 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.081905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.082103 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.082140 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.082244 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.082384 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2091,"thread_start_us":1969,"threads_started":1}
I20260925 05:07:41.082998 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.064896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.8.215.1:39177)
0925 05:07:41.073132 (+ 8236us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.073141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.073168 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:07:41.073261 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.073267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.073499 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.073683 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:41.074365 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.075751 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.080635 (+ 4884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.081702 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.081847 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.082064 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.082333 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.082684 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8078,"thread_start_us":7945,"threads_started":1}
I20260925 05:07:41.086097 16121 tablet_service.cc:1467] Processing CreateTablet for tablet b720c75e7bc3472fabcdb34052a47f19 (DEFAULT_TABLE table=test-table [id=18e06e1f90f644f28286728f672fbbe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:41.087342 16121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b720c75e7bc3472fabcdb34052a47f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:41.098899 16509 tablet_bootstrap.cc:492] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Bootstrap starting.
I20260925 05:07:41.100934 16508 tablet_bootstrap.cc:492] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Bootstrap starting.
I20260925 05:07:41.103564 16509 tablet_bootstrap.cc:654] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:41.103724 16508 tablet_bootstrap.cc:654] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:41.104780 16508 log.cc:824] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:41.105037 16509 log.cc:824] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:41.106104 16510 tablet_bootstrap.cc:492] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Bootstrap starting.
I20260925 05:07:41.110214 16510 tablet_bootstrap.cc:654] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:41.110788 16508 tablet_bootstrap.cc:492] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: No bootstrap required, opened a new log
I20260925 05:07:41.111048 16508 ts_tablet_manager.cc:1397] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 05:07:41.111452 16510 log.cc:824] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:41.111635 16509 tablet_bootstrap.cc:492] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: No bootstrap required, opened a new log
I20260925 05:07:41.111888 16509 ts_tablet_manager.cc:1397] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 05:07:41.114427 16510 tablet_bootstrap.cc:492] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: No bootstrap required, opened a new log
I20260925 05:07:41.114665 16510 ts_tablet_manager.cc:1397] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 05:07:41.118884 16508 raft_consensus.cc:348] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.119251 16508 raft_consensus.cc:374] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:41.119352 16508 raft_consensus.cc:729] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce603504d2b4ea1902c513b97272c52, State: Initialized, Role: FOLLOWER
I20260925 05:07:41.119958 16508 consensus_queue.cc:260] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.121677 16508 ts_tablet_manager.cc:1428] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 05:07:41.121914 16473 heartbeater.cc:503] Master 127.8.215.62:36215 was elected leader, sending a full tablet report...
I20260925 05:07:41.125015 16510 raft_consensus.cc:348] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.125444 16510 raft_consensus.cc:374] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:41.125561 16510 raft_consensus.cc:729] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cf57f6810e4f278a02eeae532b7258, State: Initialized, Role: FOLLOWER
I20260925 05:07:41.125866 16509 raft_consensus.cc:348] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.126276 16509 raft_consensus.cc:374] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:41.126374 16509 raft_consensus.cc:729] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc7a29b523b433b8056553dd4a77ae0, State: Initialized, Role: FOLLOWER
I20260925 05:07:41.126221 16510 consensus_queue.cc:260] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.127103 16509 consensus_queue.cc:260] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.132251 16332 heartbeater.cc:503] Master 127.8.215.62:36215 was elected leader, sending a full tablet report...
I20260925 05:07:41.134435 16510 ts_tablet_manager.cc:1428] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260925 05:07:41.137564 16509 ts_tablet_manager.cc:1428] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Time spent starting tablet: real 0.026s user 0.014s sys 0.001s
W20260925 05:07:41.234262 16477 tablet.cc:2369] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:07:41.245959 16189 tablet.cc:2369] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:41.325991 16515 raft_consensus.cc:482] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:41.326334 16515 raft_consensus.cc:504] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.327975 16515 leader_election.cc:288] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959), 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:41.334697 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.328527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:37909 (local address 127.8.215.3:39793)
0925 05:07:41.328761 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.328769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.329280 (+ 511us) server_negotiation.cc:407] Connection header received
0925 05:07:41.329467 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.329472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.329553 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.329664 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:41.330355 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.331071 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.333028 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.333764 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.333915 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.334024 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.334319 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.334438 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:07:41.334740 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.328506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:44753 (local address 127.8.215.2:34959)
0925 05:07:41.328709 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.328717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.329247 (+ 530us) server_negotiation.cc:407] Connection header received
0925 05:07:41.329469 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.329473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.329531 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.329651 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:41.330356 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.331056 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.333070 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.333823 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.333979 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.334079 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.334353 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.334475 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:07:41.334851 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.328257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39793 (local address 127.8.215.1:37909)
0925 05:07:41.329174 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:41.329189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:41.329342 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:41.329688 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:41.329692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:41.329738 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:41.330193 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.330203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.331230 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.331234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.332888 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.332900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.333997 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.334006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.334166 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.334190 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.334295 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.334420 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":56,"spinlock_wait_cycles":4224,"thread_start_us":534,"threads_started":1}
I20260925 05:07:41.335240 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.328237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:34959 (local address 127.8.215.1:44753)
0925 05:07:41.329127 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:41.329142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:41.329314 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:41.329689 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:41.329693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:41.329743 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:41.330200 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.330209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.331192 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.331199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.332919 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:41.332930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.334045 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.334051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:41.334207 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.334228 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:07:41.334328 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:07:41.334445 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":152,"spinlock_wait_cycles":7168,"thread_start_us":526,"threads_started":1}
I20260925 05:07:41.335811 16427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b720c75e7bc3472fabcdb34052a47f19" candidate_uuid: "d2cf57f6810e4f278a02eeae532b7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce603504d2b4ea1902c513b97272c52" is_pre_election: true
I20260925 05:07:41.336298 16427 raft_consensus.cc:2454] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2cf57f6810e4f278a02eeae532b7258 in term 0.
I20260925 05:07:41.336967 16077 leader_election.cc:302] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce603504d2b4ea1902c513b97272c52, d2cf57f6810e4f278a02eeae532b7258; no voters:
I20260925 05:07:41.337357 16515 raft_consensus.cc:2781] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:41.337522 16515 raft_consensus.cc:482] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:41.337646 16515 raft_consensus.cc:3037] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:41.337565 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b720c75e7bc3472fabcdb34052a47f19" candidate_uuid: "d2cf57f6810e4f278a02eeae532b7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc7a29b523b433b8056553dd4a77ae0" is_pre_election: true
I20260925 05:07:41.338083 16286 raft_consensus.cc:2454] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2cf57f6810e4f278a02eeae532b7258 in term 0.
I20260925 05:07:41.340186 16515 raft_consensus.cc:504] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.340813 16515 leader_election.cc:288] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [CANDIDATE]: Term 1 election: Requested vote from peers 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959), 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:41.341171 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b720c75e7bc3472fabcdb34052a47f19" candidate_uuid: "d2cf57f6810e4f278a02eeae532b7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc7a29b523b433b8056553dd4a77ae0"
I20260925 05:07:41.341399 16286 raft_consensus.cc:3037] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:41.341377 16427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b720c75e7bc3472fabcdb34052a47f19" candidate_uuid: "d2cf57f6810e4f278a02eeae532b7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce603504d2b4ea1902c513b97272c52"
I20260925 05:07:41.341576 16427 raft_consensus.cc:3037] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:41.344663 16286 raft_consensus.cc:2454] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2cf57f6810e4f278a02eeae532b7258 in term 1.
I20260925 05:07:41.344663 16427 raft_consensus.cc:2454] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2cf57f6810e4f278a02eeae532b7258 in term 1.
I20260925 05:07:41.345160 16077 leader_election.cc:302] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ce603504d2b4ea1902c513b97272c52, 5cc7a29b523b433b8056553dd4a77ae0, d2cf57f6810e4f278a02eeae532b7258; no voters:
I20260925 05:07:41.345496 16515 raft_consensus.cc:2781] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:41.345971 16515 raft_consensus.cc:686] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 1 LEADER]: Becoming Leader. State: Replica: d2cf57f6810e4f278a02eeae532b7258, State: Running, Role: LEADER
I20260925 05:07:41.346581 16515 consensus_queue.cc:237] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:41.352226 15219 catalog_manager.cc:5495] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1). New cstate: current_term: 1 leader_uuid: "d2cf57f6810e4f278a02eeae532b7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:41.374893 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.367699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.8.215.1:39177)
0925 05:07:41.367889 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.367895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.368147 (+ 252us) server_negotiation.cc:407] Connection header received
0925 05:07:41.368353 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.368361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.368498 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.368675 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:41.369064 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.370559 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.371973 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.372573 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.372752 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.373393 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:41.374250 (+ 857us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:41.374360 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.374524 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.374588 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 05:07:41.379138 16334 tablet.cc:2369] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:41.382876 16286 raft_consensus.cc:1261] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Refusing update from remote peer d2cf57f6810e4f278a02eeae532b7258: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:41.383004 16427 raft_consensus.cc:1261] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Refusing update from remote peer d2cf57f6810e4f278a02eeae532b7258: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:41.383945 16519 consensus_queue.cc:1035] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:41.384313 16515 consensus_queue.cc:1035] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:41.397467 16525 mvcc.cc:205] Tried to move back new op lower bound from 7333121480214953984 to 7333121480079065088. Current Snapshot: MvccSnapshot[applied={T|T < 7333121480214953984}]
I20260925 05:07:41.421505 15219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:41.421767 15219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:41.428476 15219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d648d6d50cd4c9bb2089acea54b0f6f: Sending DeleteTablet for 3 replicas of tablet b720c75e7bc3472fabcdb34052a47f19
I20260925 05:07:41.429522 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet b720c75e7bc3472fabcdb34052a47f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:41 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:36410
I20260925 05:07:41.430338 16532 tablet_replica.cc:330] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: stopping tablet replica
I20260925 05:07:41.430780 16532 raft_consensus.cc:2229] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:41.431281 16532 raft_consensus.cc:2258] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:41.431310 16121 tablet_service.cc:1514] Processing DeleteTablet for tablet b720c75e7bc3472fabcdb34052a47f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:41 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:53878
I20260925 05:07:41.431313 16407 tablet_service.cc:1514] Processing DeleteTablet for tablet b720c75e7bc3472fabcdb34052a47f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:41 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:43158
I20260925 05:07:41.433418 16532 ts_tablet_manager.cc:1905] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:41.437224 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15188
I20260925 05:07:41.438211 15250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:41.441075 16533 tablet_replica.cc:330] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: stopping tablet replica
I20260925 05:07:41.441608 16533 raft_consensus.cc:2229] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:41.442219 16533 raft_consensus.cc:2258] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:41.442852 16532 ts_tablet_manager.cc:1918] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:41.443056 16532 log.cc:1196] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/b720c75e7bc3472fabcdb34052a47f19
I20260925 05:07:41.443696 16532 ts_tablet_manager.cc:1939] T b720c75e7bc3472fabcdb34052a47f19 P 5cc7a29b523b433b8056553dd4a77ae0: Deleting consensus metadata
I20260925 05:07:41.446980 16534 tablet_replica.cc:330] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: stopping tablet replica
I20260925 05:07:41.447191 16533 ts_tablet_manager.cc:1905] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:41.447423 16534 raft_consensus.cc:2229] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:41.448052 16534 raft_consensus.cc:2258] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:41.450515 16534 ts_tablet_manager.cc:1905] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:41.454501 16533 ts_tablet_manager.cc:1918] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:41.454699 16533 log.cc:1196] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/b720c75e7bc3472fabcdb34052a47f19
I20260925 05:07:41.455220 16533 ts_tablet_manager.cc:1939] T b720c75e7bc3472fabcdb34052a47f19 P 3ce603504d2b4ea1902c513b97272c52: Deleting consensus metadata
I20260925 05:07:41.457840 16534 ts_tablet_manager.cc:1918] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:41.458014 16534 log.cc:1196] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/b720c75e7bc3472fabcdb34052a47f19
I20260925 05:07:41.458563 16534 ts_tablet_manager.cc:1939] T b720c75e7bc3472fabcdb34052a47f19 P d2cf57f6810e4f278a02eeae532b7258: Deleting consensus metadata
I20260925 05:07:41.466589 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.434989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.8.215.61:43439)
0925 05:07:41.435175 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.435182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.435250 (+ 68us) server_negotiation.cc:407] Connection header received
0925 05:07:41.435507 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.435513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.435592 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.435659 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.446702 (+ 11043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.447821 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.451047 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.451812 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.451919 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.459812 (+ 7893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.459819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.459824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.459870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.463564 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.464394 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.464400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.464404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.464563 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.465012 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.465017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.465020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.465370 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.465630 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.466018 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.466238 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:07:41.483121 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.435780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.8.215.60:41459)
0925 05:07:41.435930 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.435936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.466827 (+ 30891us) server_negotiation.cc:407] Connection header received
0925 05:07:41.467082 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.467089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.467219 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.467297 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.469332 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.470412 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.472415 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.473130 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.473253 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.476474 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.476480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.476493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.476535 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.480061 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.480873 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.480881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.480886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.481024 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.481442 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.481447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.481452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.481801 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.482042 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.482503 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.482731 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:07:41.578351 15207 catalog_manager.cc:4841] TS 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793): tablet b720c75e7bc3472fabcdb34052a47f19 (table test-table [id=18e06e1f90f644f28286728f672fbbe9]) successfully deleted
I20260925 05:07:41.579753 15206 catalog_manager.cc:4841] TS 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959): tablet b720c75e7bc3472fabcdb34052a47f19 (table test-table [id=18e06e1f90f644f28286728f672fbbe9]) successfully deleted
I20260925 05:07:41.581099 15205 catalog_manager.cc:4841] TS d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177): tablet b720c75e7bc3472fabcdb34052a47f19 (table test-table [id=18e06e1f90f644f28286728f672fbbe9]) successfully deleted
I20260925 05:07:41.581595 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15188
I20260925 05:07:41.641865 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.616945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.8.215.61:43439)
0925 05:07:41.617123 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.617129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.624173 (+ 7044us) server_negotiation.cc:407] Connection header received
0925 05:07:41.625477 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.625486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.625587 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.625685 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.629482 (+ 3797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.630646 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.632538 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.632989 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.633116 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.636265 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.636272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.636278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.636321 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.639060 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.639867 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.639872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.639876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.640001 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.640387 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.640393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.640396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.640649 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.640926 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.641235 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.641451 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:07:41.642498 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:41.617041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.8.215.60:41459)
0925 05:07:41.617257 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:41.617265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:41.625081 (+ 7816us) server_negotiation.cc:407] Connection header received
0925 05:07:41.625285 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:41.625291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:41.625376 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:41.625479 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:41.628338 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.629504 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.631232 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:41.631649 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:41.631783 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:41.635650 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:41.635656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:41.635659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:41.635698 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:41.638925 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.639614 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.639622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.639628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.639752 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:41.640819 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:41.640825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:41.640829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:41.641167 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:41.641393 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:41.641957 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 05:07:41.642164 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:07:42.409230 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.408457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:56055)
0925 05:07:42.408968 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.409075 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":305,"threads_started":1}
W20260925 05:07:42.409662 16187 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:36215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.412214 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.411354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:46967)
0925 05:07:42.411821 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.411974 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
W20260925 05:07:42.412560 16473 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:36215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.421373 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.420397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:38691)
0925 05:07:42.421019 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.421161 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":297,"threads_started":1}
W20260925 05:07:42.421736 16332 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:36215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.928880 16547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d648d6d50cd4c9bb2089acea54b0f6f)
I20260925 05:07:42.929066 16547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } }
I20260925 05:07:42.930151 16548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d648d6d50cd4c9bb2089acea54b0f6f)
I20260925 05:07:42.930486 16548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } }
I20260925 05:07:42.934633 16548 leader_election.cc:288] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215), 2aafd2c3c69c4d0897dc79edf79fdd3b (127.8.215.61:43439)
I20260925 05:07:42.935959 16547 leader_election.cc:288] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215), 22d428d011a842c693e4d23d7f4addcf (127.8.215.60:41459)
I20260925 05:07:42.941828 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.932743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50850)
0925 05:07:42.941289 (+ 8546us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.941520 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8385,"threads_started":1}
W20260925 05:07:42.942420 15992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.946650 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.930202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50834)
0925 05:07:42.946090 (+ 15888us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.946318 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15773,"mutex_wait_us":170,"thread_start_us":14059,"threads_started":1}
W20260925 05:07:42.947016 15804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.950585 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.950173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50862)
0925 05:07:42.950308 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.950433 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:07:42.950882 15804 leader_election.cc:334] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215): Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.951287 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.950228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50866)
0925 05:07:42.950766 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.950974 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":308,"threads_started":1}
W20260925 05:07:42.951512 15992 leader_election.cc:334] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215): Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.953018 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.930358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.8.215.60:41459)
0925 05:07:42.943909 (+ 13551us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:42.943919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:42.943942 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:07:42.944036 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:42.944044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:42.944142 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:42.944260 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:42.945686 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.947221 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.950562 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.952006 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.952132 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:42.952244 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:42.952608 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 05:07:42.952703 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13396,"thread_start_us":14125,"threads_started":1}
I20260925 05:07:42.953092 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.930198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:41459 (local address 127.0.0.1:51544)
0925 05:07:42.943154 (+ 12956us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.943178 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:42.943356 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:42.944862 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:42.944872 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:42.944908 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:42.945390 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:42.945408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.947437 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.947446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:42.950348 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:42.950366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.952190 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.952199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:42.952423 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:42.952465 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:07:42.952567 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:07:42.952698 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12802,"mutex_wait_us":5,"thread_start_us":15501,"threads_started":1}
I20260925 05:07:42.953802 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "22d428d011a842c693e4d23d7f4addcf" is_pre_election: true
I20260925 05:07:42.954176 16016 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aafd2c3c69c4d0897dc79edf79fdd3b in term 1.
I20260925 05:07:42.954656 15805 leader_election.cc:302] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22d428d011a842c693e4d23d7f4addcf, 2aafd2c3c69c4d0897dc79edf79fdd3b; no voters: 6d648d6d50cd4c9bb2089acea54b0f6f
I20260925 05:07:42.954983 16547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 05:07:42.955089 16547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d648d6d50cd4c9bb2089acea54b0f6f)
I20260925 05:07:42.955200 16547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:07:42.957305 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.945084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:43439 (local address 127.0.0.1:37270)
0925 05:07:42.945390 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.945412 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:42.945542 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:42.951735 (+ 6193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:42.951740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:42.951767 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:42.951981 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:42.951991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.953543 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.953549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:42.955302 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:42.955316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.956530 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.956536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:42.956715 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:42.956781 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 05:07:42.956868 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:07:42.957003 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 05:07:42.957389 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.944989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.8.215.61:43439)
0925 05:07:42.951415 (+ 6426us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:42.951422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:42.951442 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:07:42.951501 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:42.951508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:42.951582 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:42.951696 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:42.952166 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.953407 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.955456 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:42.956350 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:42.956509 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:42.956630 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:42.956890 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 05:07:42.957012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":6317,"thread_start_us":6156,"threads_started":1}
I20260925 05:07:42.957679 16547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } }
I20260925 05:07:42.958061 15828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22d428d011a842c693e4d23d7f4addcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" is_pre_election: true
I20260925 05:07:42.958458 15828 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22d428d011a842c693e4d23d7f4addcf in current term 2: Already voted for candidate 2aafd2c3c69c4d0897dc79edf79fdd3b in this term.
I20260925 05:07:42.958742 16547 leader_election.cc:288] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 2 election: Requested vote from peers 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215), 22d428d011a842c693e4d23d7f4addcf (127.8.215.60:41459)
I20260925 05:07:42.958823 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.958429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50878)
0925 05:07:42.958579 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.958662 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:07:42.958899 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "22d428d011a842c693e4d23d7f4addcf"
I20260925 05:07:42.959156 16016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:07:42.959117 15990 leader_election.cc:302] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22d428d011a842c693e4d23d7f4addcf; no voters: 2aafd2c3c69c4d0897dc79edf79fdd3b, 6d648d6d50cd4c9bb2089acea54b0f6f
I20260925 05:07:42.959824 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:42.959483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50894)
0925 05:07:42.959643 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:42.959727 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 05:07:42.959986 15804 leader_election.cc:334] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215): Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
I20260925 05:07:42.961433 16016 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aafd2c3c69c4d0897dc79edf79fdd3b in term 2.
I20260925 05:07:42.961638 16548 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:07:42.962065 15805 leader_election.cc:302] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22d428d011a842c693e4d23d7f4addcf, 2aafd2c3c69c4d0897dc79edf79fdd3b; no voters: 6d648d6d50cd4c9bb2089acea54b0f6f
I20260925 05:07:42.962378 16547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:07:42.962822 16547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [term 2 LEADER]: Becoming Leader. State: Replica: 2aafd2c3c69c4d0897dc79edf79fdd3b, State: Running, Role: LEADER
I20260925 05:07:42.963227 16547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } }
I20260925 05:07:42.965081 16557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aafd2c3c69c4d0897dc79edf79fdd3b. Latest consensus state: current_term: 2 leader_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:42.965487 16557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:42.966189 16559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:42.971444 16559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:42.972468 16559 catalog_manager.cc:1259] Loaded cluster ID: dfbfdca724914b0baa3b67d52b054438
I20260925 05:07:42.972546 16559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:42.973861 16559 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:42.974782 16559 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Loaded TSK: 0
I20260925 05:07:42.975672 16559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:07:43.027436 15817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:43.034397 15817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:43.038761 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.038380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50918)
0925 05:07:43.038545 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.038632 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 05:07:43.038934 16016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [term 2 FOLLOWER]: Refusing update from remote peer 2aafd2c3c69c4d0897dc79edf79fdd3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 05:07:43.039865 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.039422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50934)
0925 05:07:43.039589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.039670 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 05:07:43.039714 16557 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 05:07:43.040038 15804 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b -> Peer 6d648d6d50cd4c9bb2089acea54b0f6f (127.8.215.62:36215): Couldn't send request to peer 6d648d6d50cd4c9bb2089acea54b0f6f. Status: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:07:43.042523 16548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aafd2c3c69c4d0897dc79edf79fdd3b. Latest consensus state: current_term: 2 leader_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:43.042761 16548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:43.044771 16557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:43.045039 16557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:43.045064 16561 mvcc.cc:205] Tried to move back new op lower bound from 7333121487000064000 to 7333121486700421120. Current Snapshot: MvccSnapshot[applied={T|T < 7333121487000064000}]
I20260925 05:07:43.045672 16548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d648d6d50cd4c9bb2089acea54b0f6f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36215 } } peers { permanent_uuid: "2aafd2c3c69c4d0897dc79edf79fdd3b" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 43439 } attrs { promote: false } } peers { permanent_uuid: "22d428d011a842c693e4d23d7f4addcf" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 41459 } attrs { promote: false } } }
I20260925 05:07:43.046000 16548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d428d011a842c693e4d23d7f4addcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:43.062230 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.053836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:39177 (local address 127.0.0.1:43886)
0925 05:07:43.054002 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.054018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:43.054099 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:43.054837 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:43.054842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:43.054864 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:43.055051 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.055059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.057971 (+ 2912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.057978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.060102 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.060115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.061402 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.061408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.061675 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.061704 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 05:07:43.061807 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:07:43.061940 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":58}
I20260925 05:07:43.062314 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.054017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.8.215.1:39177)
0925 05:07:43.054529 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.054538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.054558 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:07:43.054625 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.054631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.054711 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.054822 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:43.055190 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.057832 (+ 2642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.060308 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.061236 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.061383 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.061478 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.061840 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.061992 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":273,"threads_started":1}
I20260925 05:07:43.064188 16121 tablet_service.cc:1467] Processing CreateTablet for tablet ea802ce228574c82ac9e533649e62d64 (DEFAULT_TABLE table=test-table [id=6e43ba55f685473999866a2f63482948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:43.064651 16121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea802ce228574c82ac9e533649e62d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:43.065125 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.054327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:34959 (local address 127.0.0.1:50232)
0925 05:07:43.054487 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.054503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:43.054625 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:43.058964 (+ 4339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:43.058969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:43.058994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:43.059340 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.059350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.060687 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.060696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.062978 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.062993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.064429 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.064435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.064604 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.064657 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 05:07:43.064753 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:07:43.064877 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260925 05:07:43.066041 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.054667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.8.215.2:34959)
0925 05:07:43.058670 (+ 4003us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.058675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.058693 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:07:43.058752 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.058756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.058823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.058951 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:43.059486 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.060551 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.063136 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.064174 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.065461 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.065564 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.065626 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.065755 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3893,"thread_start_us":3800,"threads_started":1}
I20260925 05:07:43.066712 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.055425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39793 (local address 127.0.0.1:34236)
0925 05:07:43.058391 (+ 2966us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.058407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:43.058512 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:43.059631 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:43.059636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:43.059658 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:43.060128 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.060136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.062308 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.062314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.064425 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.064441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.065884 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.065901 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.066085 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.066141 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 05:07:43.066245 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:07:43.066366 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2838,"thread_start_us":2677,"threads_started":1}
I20260925 05:07:43.066797 16266 tablet_service.cc:1467] Processing CreateTablet for tablet ea802ce228574c82ac9e533649e62d64 (DEFAULT_TABLE table=test-table [id=6e43ba55f685473999866a2f63482948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:43.067234 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea802ce228574c82ac9e533649e62d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:43.067411 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.055517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.8.215.3:39793)
0925 05:07:43.058916 (+ 3399us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.058922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.058939 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:07:43.059001 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.059006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.059078 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.060826 (+ 1748us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:43.061040 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.062139 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.064640 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.065671 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.065833 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.065952 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.066961 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.067108 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3284,"thread_start_us":4712,"threads_started":1}
I20260925 05:07:43.070238 16570 tablet_bootstrap.cc:492] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Bootstrap starting.
I20260925 05:07:43.072153 16570 tablet_bootstrap.cc:654] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:43.072425 16407 tablet_service.cc:1467] Processing CreateTablet for tablet ea802ce228574c82ac9e533649e62d64 (DEFAULT_TABLE table=test-table [id=6e43ba55f685473999866a2f63482948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:43.072890 16407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea802ce228574c82ac9e533649e62d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:43.074704 16570 tablet_bootstrap.cc:492] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: No bootstrap required, opened a new log
I20260925 05:07:43.074906 16570 ts_tablet_manager.cc:1397] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 05:07:43.076014 16570 raft_consensus.cc:348] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.076355 16570 raft_consensus.cc:374] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:43.076478 16570 raft_consensus.cc:729] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cf57f6810e4f278a02eeae532b7258, State: Initialized, Role: FOLLOWER
I20260925 05:07:43.076810 16570 consensus_queue.cc:260] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.077872 16570 ts_tablet_manager.cc:1428] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:43.081624 16188 heartbeater.cc:503] Master 127.8.215.61:43439 was elected leader, sending a full tablet report...
I20260925 05:07:43.082429 16572 tablet_bootstrap.cc:492] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Bootstrap starting.
I20260925 05:07:43.085156 16575 tablet_bootstrap.cc:492] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Bootstrap starting.
I20260925 05:07:43.085470 16572 tablet_bootstrap.cc:654] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:43.085958 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.078086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:58437)
0925 05:07:43.085462 (+ 7376us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.085611 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7189,"thread_start_us":7088,"threads_started":1}
I20260925 05:07:43.087142 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.086720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:50599)
0925 05:07:43.086912 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.087001 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:07:43.087512 16575 tablet_bootstrap.cc:654] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:43.087824 16572 tablet_bootstrap.cc:492] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: No bootstrap required, opened a new log
I20260925 05:07:43.088006 16572 ts_tablet_manager.cc:1397] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 05:07:43.088665 16572 raft_consensus.cc:348] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.088925 16572 raft_consensus.cc:374] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:43.089021 16572 raft_consensus.cc:729] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc7a29b523b433b8056553dd4a77ae0, State: Initialized, Role: FOLLOWER
I20260925 05:07:43.089265 16572 consensus_queue.cc:260] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.089816 16575 tablet_bootstrap.cc:492] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: No bootstrap required, opened a new log
I20260925 05:07:43.089972 16575 ts_tablet_manager.cc:1397] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:07:43.090031 16572 ts_tablet_manager.cc:1428] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:07:43.090942 16575 raft_consensus.cc:348] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.093201 16575 raft_consensus.cc:374] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:43.093353 16575 raft_consensus.cc:729] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce603504d2b4ea1902c513b97272c52, State: Initialized, Role: FOLLOWER
I20260925 05:07:43.093673 16575 consensus_queue.cc:260] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.094732 16575 ts_tablet_manager.cc:1428] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:07:43.096315 16333 heartbeater.cc:503] Master 127.8.215.61:43439 was elected leader, sending a full tablet report...
I20260925 05:07:43.096498 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.095431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:43857)
0925 05:07:43.096039 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.096201 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"threads_started":1}
I20260925 05:07:43.097326 16474 heartbeater.cc:503] Master 127.8.215.61:43439 was elected leader, sending a full tablet report...
I20260925 05:07:43.100200 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.099664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:33297)
0925 05:07:43.099864 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.100007 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:07:43.101599 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.090344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:35847)
0925 05:07:43.101261 (+ 10917us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.101379 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10783,"thread_start_us":10637,"threads_started":1}
I20260925 05:07:43.102560 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.102231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:55005)
0925 05:07:43.102379 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.102460 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:07:43.349668 16578 raft_consensus.cc:482] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:43.349928 16578 raft_consensus.cc:504] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.351444 16578 leader_election.cc:288] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177), 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:43.358577 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.351824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50957 (local address 127.8.215.1:39177)
0925 05:07:43.352010 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.352017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.352711 (+ 694us) server_negotiation.cc:407] Connection header received
0925 05:07:43.352893 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.352900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.352988 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.353088 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:43.353626 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.354458 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.356763 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.357524 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.357694 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.357800 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.358164 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.358299 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:07:43.358785 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.351886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:44853 (local address 127.8.215.3:39793)
0925 05:07:43.352083 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.352087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.352258 (+ 171us) server_negotiation.cc:407] Connection header received
0925 05:07:43.352512 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.352519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.352601 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.352698 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:43.353316 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.354137 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.356633 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.357885 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.358012 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.358107 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.358430 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.358552 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:07:43.358847 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.351722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39793 (local address 127.8.215.2:44853)
0925 05:07:43.352126 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.352152 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:43.352352 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:43.352772 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:43.352777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:43.352838 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:43.353161 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.353173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.354283 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.354288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.356459 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.356474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.358101 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.358109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.358281 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.358311 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:07:43.358400 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:07:43.358530 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":26}
I20260925 05:07:43.359522 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.351844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:39177 (local address 127.8.215.2:50957)
0925 05:07:43.352578 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.352595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:43.352752 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:43.353120 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:43.353124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:43.353146 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:43.353481 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.353490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.354596 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.354601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.356598 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:43.356610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.357749 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.357756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:43.357934 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.357988 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 05:07:43.359031 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0925 05:07:43.359161 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":1,"thread_start_us":415,"threads_started":1}
I20260925 05:07:43.359488 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea802ce228574c82ac9e533649e62d64" candidate_uuid: "5cc7a29b523b433b8056553dd4a77ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce603504d2b4ea1902c513b97272c52" is_pre_election: true
I20260925 05:07:43.359798 16426 raft_consensus.cc:2454] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cc7a29b523b433b8056553dd4a77ae0 in term 0.
I20260925 05:07:43.360351 16222 leader_election.cc:302] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce603504d2b4ea1902c513b97272c52, 5cc7a29b523b433b8056553dd4a77ae0; no voters:
I20260925 05:07:43.360359 16141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea802ce228574c82ac9e533649e62d64" candidate_uuid: "5cc7a29b523b433b8056553dd4a77ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cf57f6810e4f278a02eeae532b7258" is_pre_election: true
I20260925 05:07:43.360802 16578 raft_consensus.cc:2781] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:43.360805 16141 raft_consensus.cc:2454] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cc7a29b523b433b8056553dd4a77ae0 in term 0.
I20260925 05:07:43.360927 16578 raft_consensus.cc:482] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:43.361020 16578 raft_consensus.cc:3037] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:43.363565 16578 raft_consensus.cc:504] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.364185 16578 leader_election.cc:288] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [CANDIDATE]: Term 1 election: Requested vote from peers d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177), 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793)
I20260925 05:07:43.364567 16141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea802ce228574c82ac9e533649e62d64" candidate_uuid: "5cc7a29b523b433b8056553dd4a77ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cf57f6810e4f278a02eeae532b7258"
I20260925 05:07:43.364769 16141 raft_consensus.cc:3037] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:43.364660 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea802ce228574c82ac9e533649e62d64" candidate_uuid: "5cc7a29b523b433b8056553dd4a77ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce603504d2b4ea1902c513b97272c52"
I20260925 05:07:43.364925 16426 raft_consensus.cc:3037] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:43.366811 16141 raft_consensus.cc:2454] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cc7a29b523b433b8056553dd4a77ae0 in term 1.
I20260925 05:07:43.367219 16220 leader_election.cc:302] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc7a29b523b433b8056553dd4a77ae0, d2cf57f6810e4f278a02eeae532b7258; no voters:
I20260925 05:07:43.367507 16578 raft_consensus.cc:2781] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:43.367601 16426 raft_consensus.cc:2454] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cc7a29b523b433b8056553dd4a77ae0 in term 1.
I20260925 05:07:43.368064 16578 raft_consensus.cc:686] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 5cc7a29b523b433b8056553dd4a77ae0, State: Running, Role: LEADER
I20260925 05:07:43.368494 16578 consensus_queue.cc:237] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } }
I20260925 05:07:43.369055 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.368659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:49543)
0925 05:07:43.368835 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.368934 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 05:07:43.371943 15817 catalog_manager.cc:5495] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "5cc7a29b523b433b8056553dd4a77ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cc7a29b523b433b8056553dd4a77ae0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 34959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:43.428366 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.423262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.8.215.2:34959)
0925 05:07:43.423481 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.423487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.423506 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:07:43.423714 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.423718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.423801 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.423954 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:43.424318 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.425046 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.426307 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.426708 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.426841 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.427248 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:43.427777 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:43.427930 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.428024 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.428101 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:07:43.432410 16426 raft_consensus.cc:1261] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Refusing update from remote peer 5cc7a29b523b433b8056553dd4a77ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:43.432687 16141 raft_consensus.cc:1261] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Refusing update from remote peer 5cc7a29b523b433b8056553dd4a77ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:43.433189 16583 consensus_queue.cc:1035] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce603504d2b4ea1902c513b97272c52" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:43.433517 16578 consensus_queue.cc:1035] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cf57f6810e4f278a02eeae532b7258" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:43.435942 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.435307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:37375)
0925 05:07:43.435548 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.435688 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 05:07:43.437393 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.436625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:34789)
0925 05:07:43.437108 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.437224 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
I20260925 05:07:43.441780 16586 mvcc.cc:205] Tried to move back new op lower bound from 7333121488612442112 to 7333121488361525248. Current Snapshot: MvccSnapshot[applied={T|T < 7333121488612442112}]
I20260925 05:07:43.442665 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.442202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.3:47729)
0925 05:07:43.442405 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.442493 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 05:07:43.443663 16588 mvcc.cc:205] Tried to move back new op lower bound from 7333121488612442112 to 7333121488361525248. Current Snapshot: MvccSnapshot[applied={T|T < 7333121488612442112}]
I20260925 05:07:43.444993 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.444321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.2:41021)
0925 05:07:43.444744 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.444837 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270}
I20260925 05:07:43.445212 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.444716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.8.215.1:60223)
0925 05:07:43.444959 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.445055 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 05:07:43.455571 15818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:43.455839 15818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:43.461431 15818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2aafd2c3c69c4d0897dc79edf79fdd3b: Sending DeleteTablet for 3 replicas of tablet ea802ce228574c82ac9e533649e62d64
I20260925 05:07:43.462558 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet ea802ce228574c82ac9e533649e62d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:43 UTC) from {username='oryx', principal='oryx/127.8.215.61@KRBTEST.COM'} at 127.0.0.1:50232
I20260925 05:07:43.462869 16121 tablet_service.cc:1514] Processing DeleteTablet for tablet ea802ce228574c82ac9e533649e62d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:43 UTC) from {username='oryx', principal='oryx/127.8.215.61@KRBTEST.COM'} at 127.0.0.1:43886
I20260925 05:07:43.463433 16595 tablet_replica.cc:330] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: stopping tablet replica
I20260925 05:07:43.463640 16595 raft_consensus.cc:2229] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:43.463688 16407 tablet_service.cc:1514] Processing DeleteTablet for tablet ea802ce228574c82ac9e533649e62d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:43 UTC) from {username='oryx', principal='oryx/127.8.215.61@KRBTEST.COM'} at 127.0.0.1:34236
I20260925 05:07:43.464131 16595 raft_consensus.cc:2258] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:43.465312 16595 ts_tablet_manager.cc:1905] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:43.465615 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 16043
I20260925 05:07:43.472236 16595 ts_tablet_manager.cc:1918] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:43.472834 16595 log.cc:1196] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/ea802ce228574c82ac9e533649e62d64
I20260925 05:07:43.473402 16595 ts_tablet_manager.cc:1939] T ea802ce228574c82ac9e533649e62d64 P 5cc7a29b523b433b8056553dd4a77ae0: Deleting consensus metadata
I20260925 05:07:43.473930 16597 tablet_replica.cc:330] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: stopping tablet replica
I20260925 05:07:43.474236 16597 raft_consensus.cc:2229] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:43.474596 16597 raft_consensus.cc:2258] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:43.474809 15804 catalog_manager.cc:4841] TS 5cc7a29b523b433b8056553dd4a77ae0 (127.8.215.2:34959): tablet ea802ce228574c82ac9e533649e62d64 (table test-table [id=6e43ba55f685473999866a2f63482948]) successfully deleted
I20260925 05:07:43.477349 16597 ts_tablet_manager.cc:1905] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:43.480458 16596 tablet_replica.cc:330] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: stopping tablet replica
I20260925 05:07:43.480640 16596 raft_consensus.cc:2229] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:43.480952 16596 raft_consensus.cc:2258] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:43.482111 16596 ts_tablet_manager.cc:1905] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:43.484280 16597 ts_tablet_manager.cc:1918] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:43.484419 16597 log.cc:1196] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/ea802ce228574c82ac9e533649e62d64
I20260925 05:07:43.485209 16597 ts_tablet_manager.cc:1939] T ea802ce228574c82ac9e533649e62d64 P 3ce603504d2b4ea1902c513b97272c52: Deleting consensus metadata
I20260925 05:07:43.486429 15805 catalog_manager.cc:4841] TS 3ce603504d2b4ea1902c513b97272c52 (127.8.215.3:39793): tablet ea802ce228574c82ac9e533649e62d64 (table test-table [id=6e43ba55f685473999866a2f63482948]) successfully deleted
I20260925 05:07:43.488960 16596 ts_tablet_manager.cc:1918] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:43.489097 16596 log.cc:1196] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/ea802ce228574c82ac9e533649e62d64
I20260925 05:07:43.489575 16596 ts_tablet_manager.cc:1939] T ea802ce228574c82ac9e533649e62d64 P d2cf57f6810e4f278a02eeae532b7258: Deleting consensus metadata
I20260925 05:07:43.490437 15803 catalog_manager.cc:4841] TS d2cf57f6810e4f278a02eeae532b7258 (127.8.215.1:39177): tablet ea802ce228574c82ac9e533649e62d64 (table test-table [id=6e43ba55f685473999866a2f63482948]) successfully deleted
I20260925 05:07:43.491945 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.491506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50950)
0925 05:07:43.491705 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.491811 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:07:43.493043 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.492592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36215 (local address 127.0.0.1:50958)
0925 05:07:43.492807 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:43.492889 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 05:07:43.497987 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.468570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.8.215.1:39177)
0925 05:07:43.468822 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.468831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.479725 (+ 10894us) server_negotiation.cc:407] Connection header received
0925 05:07:43.479886 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.479892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.479980 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.480091 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:43.482704 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.483955 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.486127 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.486806 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.486909 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.490656 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:43.490674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:43.490694 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:43.490731 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:43.494937 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:43.495750 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:43.495757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:43.495759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:43.495913 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:43.496295 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:43.496298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:43.496300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:43.496645 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:43.496943 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.497410 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.497602 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:07:43.498678 16181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:43.624166 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 16043
I20260925 05:07:43.654394 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 16198
I20260925 05:07:43.670421 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.655486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.8.215.2:34959)
0925 05:07:43.655693 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.655700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.655719 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:07:43.655786 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.655793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.655906 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.656031 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:43.658072 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.659258 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.660705 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.661398 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.661534 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.663892 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:43.663906 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:43.663925 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:43.663964 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:43.667823 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:43.668421 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:43.668429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:43.668433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:43.668596 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:43.669003 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:43.669007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:43.669011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:43.669436 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:43.669710 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.669867 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.670057 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:07:43.671283 16326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:43.785600 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 16198
I20260925 05:07:43.805027 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 16344
I20260925 05:07:43.817421 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:43.805740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.8.215.3:39793)
0925 05:07:43.805896 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:43.805900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:43.805946 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:07:43.806104 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:43.806108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:43.806167 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:43.806250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:43.807659 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.808448 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.809749 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:43.810150 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:43.810241 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:43.812560 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:43.812573 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:43.812589 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:43.812616 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:43.815131 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:43.815733 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:43.815739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:43.815742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:43.815846 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:43.816141 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:43.816144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:43.816146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:43.816468 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:43.816699 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:43.817010 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 05:07:43.817168 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:07:43.818039 16467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:43.911659 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 16344
I20260925 05:07:43.930040 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15681
I20260925 05:07:43.931028 15848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:44.050787 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15681
W20260925 05:07:44.072587 15990 connection.cc:535] server connection from 127.0.0.1:51544 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:07:44.072651 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 15782
I20260925 05:07:44.073761 16036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:44.185879 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 15782
2026-09-25T05:07:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9824 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:44 dist-test-slave-wpdm krb5kdc[16606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:44 dist-test-slave-wpdm krb5kdc[16606](info): set up 2 sockets
Sep 25 05:07:44 dist-test-slave-wpdm krb5kdc[16606](info): commencing operation
krb5kdc: starting...
W20260925 05:07:45.330255 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:45 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab.
Entry for principal oryx/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab.
I20260925 05:07:45.556510 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44293
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab
--principal=oryx/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:45.881872 16619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:45.882148 16619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:45.882237 16619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:45.889330 16619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:45.889424 16619 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:07:45.889482 16619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:45.889542 16619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:45.889590 16619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:45.889649 16619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:45.899225 16619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:44293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.62.keytab
--principal=oryx/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:45.900379 16619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:45.901615 16619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:45.913190 16624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:45.913249 16625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:45.913269 16627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:45.913988 16619 server_base.cc:1034] running on GCE node
I20260925 05:07:45.914726 16619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:45.916072 16619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:45.917306 16619 hybrid_clock.cc:648] HybridClock initialized: now 1790312865917238 us; error 49 us; skew 500 ppm
I20260925 05:07:45.917619 16619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:45 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312865, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:45.920665 16619 init.cc:377] Logged in from keytab as oryx/127.8.215.62@KRBTEST.COM (short username oryx)
I20260925 05:07:45.922056 16619 webserver.cc:460] Webserver started at http://127.8.215.62:37403/ using document root <none> and password file <none>
I20260925 05:07:45.922755 16619 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:45.922884 16619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:45.923205 16619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:45.926133 16619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "aaf7be440fb64604814d8621674c72d4"
format_stamp: "Formatted at 2026-09-25 05:07:45 on dist-test-slave-wpdm"
server_key: "7a8a9d40754f9600f5e1c5b8e6f2a4e8"
server_key_iv: "8d6521130a1a20af7cac3a34faf1ca9b"
server_key_version: "encryptionkey@0"
I20260925 05:07:45.926905 16619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "aaf7be440fb64604814d8621674c72d4"
format_stamp: "Formatted at 2026-09-25 05:07:45 on dist-test-slave-wpdm"
server_key: "7a8a9d40754f9600f5e1c5b8e6f2a4e8"
server_key_iv: "8d6521130a1a20af7cac3a34faf1ca9b"
server_key_version: "encryptionkey@0"
I20260925 05:07:45.932538 16619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:07:45.936199 16634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:45.937529 16619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:45.937713 16619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "aaf7be440fb64604814d8621674c72d4"
format_stamp: "Formatted at 2026-09-25 05:07:45 on dist-test-slave-wpdm"
server_key: "7a8a9d40754f9600f5e1c5b8e6f2a4e8"
server_key_iv: "8d6521130a1a20af7cac3a34faf1ca9b"
server_key_version: "encryptionkey@0"
I20260925 05:07:45.937927 16619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:45.956099 16619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:45.960467 16619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:45.960764 16619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:45.983299 16619 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:44293
I20260925 05:07:45.983317 16695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:44293 every 8 connection(s)
I20260925 05:07:45.984946 16619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:45.986009 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 16619
I20260925 05:07:45.986284 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:45.986996 9052 external_mini_cluster.cc:1428] Setting key 50a0b76a5f65bc2adfcbef92ccd88ec2
I20260925 05:07:45.989439 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:07:46 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:46.002774 16696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Bootstrap starting.
I20260925 05:07:46.005923 16696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:46.007285 16696 log.cc:824] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:46.010226 16696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: No bootstrap required, opened a new log
I20260925 05:07:46.012180 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:45.989240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.8.215.62:44293)
0925 05:07:45.990173 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:45.990185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:45.990231 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:07:45.992033 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:45.992056 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:45.992489 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:45.993047 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:45.994791 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:45.996469 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:45.998213 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:45.999031 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:45.999203 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:46.004083 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:46.004097 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:46.004112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:46.004168 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:46.008270 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:46.009017 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:46.009024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:46.009030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:46.009200 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:46.009582 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:46.009591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:46.009593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:46.010062 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:46.010366 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:46.010816 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 05:07:46.011189 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":469,"threads_started":1}
I20260925 05:07:46.018347 16696 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf7be440fb64604814d8621674c72d4" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44293 } }
I20260925 05:07:46.018764 16696 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:46.018873 16696 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf7be440fb64604814d8621674c72d4, State: Initialized, Role: FOLLOWER
I20260925 05:07:46.019528 16696 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf7be440fb64604814d8621674c72d4" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44293 } }
I20260925 05:07:46.019739 16696 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:46.019832 16696 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:46.019951 16696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:46.022235 16696 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf7be440fb64604814d8621674c72d4" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44293 } }
I20260925 05:07:46.022784 16696 leader_election.cc:302] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf7be440fb64604814d8621674c72d4; no voters:
I20260925 05:07:46.023331 16696 leader_election.cc:288] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:46.023451 16701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:46.024201 16701 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [term 1 LEADER]: Becoming Leader. State: Replica: aaf7be440fb64604814d8621674c72d4, State: Running, Role: LEADER
I20260925 05:07:46.024708 16701 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf7be440fb64604814d8621674c72d4" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44293 } }
I20260925 05:07:46.025455 16696 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:46.027721 16703 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf7be440fb64604814d8621674c72d4. Latest consensus state: current_term: 1 leader_uuid: "aaf7be440fb64604814d8621674c72d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf7be440fb64604814d8621674c72d4" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44293 } } }
I20260925 05:07:46.027709 16702 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaf7be440fb64604814d8621674c72d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf7be440fb64604814d8621674c72d4" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44293 } } }
I20260925 05:07:46.028273 16703 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:46.028370 16702 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:46.034305 16709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:46.040149 16709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:46.048452 16709 catalog_manager.cc:1347] Generated new cluster ID: ff3ba07beaaf405080f8ac70098181c5
I20260925 05:07:46.048563 16709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:46.088569 16709 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:46.090457 16709 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:46.098448 16709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Generated new TSK 0
I20260925 05:07:46.099466 16709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab.
Entry for principal oryx/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab.
I20260925 05:07:46.164634 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44293
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:46.498742 16722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:46.499023 16722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:46.499126 16722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:46.506376 16722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:46.506491 16722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:46.506600 16722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:07:46.506685 16722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:46.516714 16722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:46.518023 16722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:46.519371 16722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:46.533129 16728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:46.533250 16730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:46.533211 16727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:46.534183 16722 server_base.cc:1034] running on GCE node
I20260925 05:07:46.534875 16722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:46.535880 16722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:46.537113 16722 hybrid_clock.cc:648] HybridClock initialized: now 1790312866537032 us; error 58 us; skew 500 ppm
I20260925 05:07:46.537451 16722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:46 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312866, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:46.544479 16722 init.cc:377] Logged in from keytab as oryx/127.8.215.1@KRBTEST.COM (short username oryx)
I20260925 05:07:46.546046 16722 webserver.cc:460] Webserver started at http://127.8.215.1:37597/ using document root <none> and password file <none>
I20260925 05:07:46.546772 16722 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:46.546901 16722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:46.547220 16722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:46.550154 16722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
format_stamp: "Formatted at 2026-09-25 05:07:46 on dist-test-slave-wpdm"
server_key: "eb111e37b7b65b860a22d996781ae149"
server_key_iv: "ee84c1b29ddc035d215fff3f39eee433"
server_key_version: "encryptionkey@0"
I20260925 05:07:46.550937 16722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
format_stamp: "Formatted at 2026-09-25 05:07:46 on dist-test-slave-wpdm"
server_key: "eb111e37b7b65b860a22d996781ae149"
server_key_iv: "ee84c1b29ddc035d215fff3f39eee433"
server_key_version: "encryptionkey@0"
I20260925 05:07:46.556046 16722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 05:07:46.559536 16737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:46.561036 16722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:46.561229 16722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
format_stamp: "Formatted at 2026-09-25 05:07:46 on dist-test-slave-wpdm"
server_key: "eb111e37b7b65b860a22d996781ae149"
server_key_iv: "ee84c1b29ddc035d215fff3f39eee433"
server_key_version: "encryptionkey@0"
I20260925 05:07:46.561470 16722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:46.603318 16722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:46.607741 16722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:46.608023 16722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:46.610725 16722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:46.610831 16722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:46.611016 16722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:46.611114 16722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:07:46 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312866, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:46.634013 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:46.611184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:52399 (local address 127.8.215.62:44293)
0925 05:07:46.611621 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:46.611627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:46.612915 (+ 1288us) server_negotiation.cc:407] Connection header received
0925 05:07:46.615109 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:46.615115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:46.615204 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:46.615322 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:46.617685 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.618969 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.621563 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.622196 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.622333 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:46.626641 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:46.626650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:46.626657 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:46.626700 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:46.630003 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:46.630945 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:46.630951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:46.630955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:46.631134 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:46.631832 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:46.631837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:46.631840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:46.632238 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:46.632419 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:46.633444 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0925 05:07:46.633667 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":218,"threads_started":1}
I20260925 05:07:46.635288 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:46.611865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.8.215.1:52399)
0925 05:07:46.612751 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:46.612799 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:46.614837 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:46.615615 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:46.615631 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:46.616570 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:46.617437 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:46.617469 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.619198 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.619204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:46.621421 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:46.621432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.622444 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.622451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:46.622679 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:46.623441 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:46.623465 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:46.626297 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:46.630171 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:46.630177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:46.630191 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:46.630811 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:46.631285 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:46.631297 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:46.631304 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:46.631646 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:46.632420 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:46.632436 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:46.632971 (+ 535us) client_negotiation.cc:769] Sending connection context
0925 05:07:46.633346 (+ 375us) client_negotiation.cc:240] Negotiation successful
0925 05:07:46.633897 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":382,"threads_started":1}
I20260925 05:07:46.656100 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:46.641505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.8.215.1:52583)
0925 05:07:46.642464 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:46.642499 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:46.642657 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:46.642976 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:46.642981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:46.643484 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:46.643923 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:46.643939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.645442 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.645447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:46.646999 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:46.647011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.647734 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.647741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:46.647885 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:46.648496 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:46.648513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:46.649537 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:46.653069 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:46.653074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:46.653078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:46.653569 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:46.654036 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:46.654052 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:46.654056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:46.654243 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:46.654878 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:46.654883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:46.655181 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 05:07:46.655368 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 05:07:46.655767 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":651,"threads_started":1}
I20260925 05:07:46.656646 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:46.642091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:52583 (local address 127.8.215.62:44293)
0925 05:07:46.642272 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:46.642278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:46.642591 (+ 313us) server_negotiation.cc:407] Connection header received
0925 05:07:46.642774 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:46.642778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:46.642845 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:46.642953 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:46.644140 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.645293 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.647132 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:46.647550 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:46.647675 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:46.649726 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:46.649732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:46.649736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:46.649799 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:46.652936 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:46.653717 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:46.653722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:46.653726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:46.653887 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:46.654357 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:46.654363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:46.654367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:46.654770 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:46.654923 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:46.655858 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 05:07:46.656209 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:07:46.661805 16722 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:46193
I20260925 05:07:46.661842 16857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:46193 every 8 connection(s)
I20260925 05:07:46.662983 16722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:46.664893 16858 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44293
I20260925 05:07:46.665129 16858 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:46.665849 16858 heartbeater.cc:511] Master 127.8.215.62:44293 requested a full tablet report, sending...
I20260925 05:07:46.667280 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 16722
I20260925 05:07:46.667457 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:46.667973 16651 ts_manager.cc:194] Registered new tserver with Master: 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193)
I20260925 05:07:46.668104 9052 external_mini_cluster.cc:1428] Setting key c13b341d9d9c71ac2008f3bc5230cb63
I20260925 05:07:46.669724 16651 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:52399
WARNING: no policy specified for oryx/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab.
Entry for principal oryx/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab.
I20260925 05:07:46.717680 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44293
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:47.047793 16862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:47.048058 16862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:47.048151 16862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:47.055301 16862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:47.055428 16862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:47.055511 16862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:07:47.055583 16862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:47.065304 16862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:47.066550 16862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:47.067822 16862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:47.080822 16870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:47.080881 16868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:47.080905 16867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:47.082151 16862 server_base.cc:1034] running on GCE node
I20260925 05:07:47.082746 16862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:47.083689 16862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:47.084928 16862 hybrid_clock.cc:648] HybridClock initialized: now 1790312867084860 us; error 44 us; skew 500 ppm
I20260925 05:07:47.085247 16862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312867, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:47.088483 16862 init.cc:377] Logged in from keytab as oryx/127.8.215.2@KRBTEST.COM (short username oryx)
I20260925 05:07:47.089923 16862 webserver.cc:460] Webserver started at http://127.8.215.2:40487/ using document root <none> and password file <none>
I20260925 05:07:47.090551 16862 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:47.090636 16862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:47.090894 16862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:47.093730 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "b669f9f9a9154434b3af3cdc4ce12067"
format_stamp: "Formatted at 2026-09-25 05:07:47 on dist-test-slave-wpdm"
server_key: "3149a78ad138584522232a841382897c"
server_key_iv: "cb4d402a5e2f231536f70a075db57600"
server_key_version: "encryptionkey@0"
I20260925 05:07:47.094471 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "b669f9f9a9154434b3af3cdc4ce12067"
format_stamp: "Formatted at 2026-09-25 05:07:47 on dist-test-slave-wpdm"
server_key: "3149a78ad138584522232a841382897c"
server_key_iv: "cb4d402a5e2f231536f70a075db57600"
server_key_version: "encryptionkey@0"
I20260925 05:07:47.099694 16862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:07:47.103113 16877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:47.104444 16862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:47.104568 16862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "b669f9f9a9154434b3af3cdc4ce12067"
format_stamp: "Formatted at 2026-09-25 05:07:47 on dist-test-slave-wpdm"
server_key: "3149a78ad138584522232a841382897c"
server_key_iv: "cb4d402a5e2f231536f70a075db57600"
server_key_version: "encryptionkey@0"
I20260925 05:07:47.104875 16862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:47.128293 16862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:47.132674 16862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:47.132962 16862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:47.135655 16862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:47.135737 16862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:47.135901 16862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:47.135960 16862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312867, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:47.158957 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.136364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:44977 (local address 127.8.215.62:44293)
0925 05:07:47.136577 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.136582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.138119 (+ 1537us) server_negotiation.cc:407] Connection header received
0925 05:07:47.140151 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.140160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.140251 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.140376 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:47.142777 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.144095 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.146609 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.147395 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.147557 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.152021 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:47.152032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:47.152039 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:47.152094 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:47.155363 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.156297 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.156302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.156306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.156459 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.157119 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.157126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.157131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.157515 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:47.157687 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.158385 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.158599 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:07:47.160035 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.137038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.8.215.2:44977)
0925 05:07:47.137943 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.137991 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.139848 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.140762 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.140774 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.141653 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:47.142562 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.142599 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.144282 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.144288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.146484 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.146499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.147774 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.147784 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.148035 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.148828 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:47.148868 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:47.151759 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:47.155504 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.155524 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.155546 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.156140 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.156641 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.156651 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.156654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.156966 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.157653 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:47.157661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:47.158023 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.158282 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.158811 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":430,"threads_started":1}
I20260925 05:07:47.184079 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.166379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:44801 (local address 127.8.215.62:44293)
0925 05:07:47.166568 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.166575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.167226 (+ 651us) server_negotiation.cc:407] Connection header received
0925 05:07:47.167412 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.167416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.167471 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.167552 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:47.169048 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.170366 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.173042 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.173815 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.173936 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.177133 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:47.177140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:47.177147 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:47.177200 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:47.180903 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.181860 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.181867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.181870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.181995 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.182488 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.182493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.182496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.182970 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:47.183137 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.183531 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.183734 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:07:47.184201 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.166332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.8.215.2:44801)
0925 05:07:47.167102 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.167131 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.167297 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.167607 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.167614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.168353 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:47.168851 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.168867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.170605 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.170611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.172841 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.172862 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.174284 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.174292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.174536 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.175473 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:47.175506 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:47.176889 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:47.181071 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.181081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.181086 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.181657 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.182176 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.182183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.182189 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.182366 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.183104 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:47.183112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:47.183287 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.183421 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.183732 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":396,"threads_started":1}
I20260925 05:07:47.189154 16862 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:37501
I20260925 05:07:47.189191 16996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:37501 every 8 connection(s)
I20260925 05:07:47.190352 16862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:47.192262 16997 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44293
I20260925 05:07:47.192508 16997 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:47.193231 16997 heartbeater.cc:511] Master 127.8.215.62:44293 requested a full tablet report, sending...
I20260925 05:07:47.194478 16651 ts_manager.cc:194] Registered new tserver with Master: b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:47.195392 16651 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:44977
I20260925 05:07:47.200418 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 16862
I20260925 05:07:47.200592 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:47.201036 9052 external_mini_cluster.cc:1428] Setting key 1b638da0fb12726f080900ae39a8a356
WARNING: no policy specified for oryx/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab.
Entry for principal oryx/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab.
I20260925 05:07:47.248842 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44293
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:47.565655 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:47.565985 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:47.566180 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:47.573354 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:47.573453 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:47.573608 17001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:07:47.573746 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:47.583309 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.215.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:47.584595 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:47.585987 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:47.599356 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:47.599433 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:47.599453 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:47.600297 17001 server_base.cc:1034] running on GCE node
I20260925 05:07:47.600965 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:47.601894 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:47.603137 17001 hybrid_clock.cc:648] HybridClock initialized: now 1790312867603080 us; error 43 us; skew 500 ppm
I20260925 05:07:47.603458 17001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312867, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:47.606629 17001 init.cc:377] Logged in from keytab as oryx/127.8.215.3@KRBTEST.COM (short username oryx)
I20260925 05:07:47.608148 17001 webserver.cc:460] Webserver started at http://127.8.215.3:34609/ using document root <none> and password file <none>
I20260925 05:07:47.608915 17001 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:47.609041 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:47.609359 17001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:47.612182 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "bcc67742601a494db2223f374c1e7b05"
format_stamp: "Formatted at 2026-09-25 05:07:47 on dist-test-slave-wpdm"
server_key: "c8a22646bb4693a50e5ff22ab9f7c9c4"
server_key_iv: "018dc1dae56f284986100045553bee8a"
server_key_version: "encryptionkey@0"
I20260925 05:07:47.612967 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "bcc67742601a494db2223f374c1e7b05"
format_stamp: "Formatted at 2026-09-25 05:07:47 on dist-test-slave-wpdm"
server_key: "c8a22646bb4693a50e5ff22ab9f7c9c4"
server_key_iv: "018dc1dae56f284986100045553bee8a"
server_key_version: "encryptionkey@0"
I20260925 05:07:47.617833 17001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:07:47.621594 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:47.622831 17001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:47.623005 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "bcc67742601a494db2223f374c1e7b05"
format_stamp: "Formatted at 2026-09-25 05:07:47 on dist-test-slave-wpdm"
server_key: "c8a22646bb4693a50e5ff22ab9f7c9c4"
server_key_iv: "018dc1dae56f284986100045553bee8a"
server_key_version: "encryptionkey@0"
I20260925 05:07:47.623221 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:47.642132 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:47.646253 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:47.646560 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:47.649492 17001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:47.649619 17001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:47.649804 17001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:47.649892 17001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312867, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:47.672284 16858 heartbeater.cc:503] Master 127.8.215.62:44293 was elected leader, sending a full tablet report...
I20260925 05:07:47.673786 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.649720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:51829 (local address 127.8.215.62:44293)
0925 05:07:47.649901 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.649907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.651356 (+ 1449us) server_negotiation.cc:407] Connection header received
0925 05:07:47.653395 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.653406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.653510 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.653638 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:47.655396 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.656712 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.659381 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.659940 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.660083 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.665252 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:47.665260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:47.665266 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:47.665308 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:47.669239 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.670405 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.670413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.670417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.670603 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.671226 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.671234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.671240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.671658 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:47.671794 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.673033 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.673313 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:07:47.676424 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.650312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.8.215.3:51829)
0925 05:07:47.651167 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.651214 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.653086 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.653831 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.653839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.654339 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:47.655124 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.655141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.656991 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.656999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.659221 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.659231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.660187 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.660194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.660494 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.661327 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:47.661360 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:47.664925 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:47.669392 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.669405 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.669430 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.670271 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.670743 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.670753 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.670758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.671063 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.671815 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:47.671826 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:47.672301 (+ 475us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.673985 (+ 1684us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.674573 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":375,"threads_started":1}
I20260925 05:07:47.699147 17001 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:44279
I20260925 05:07:47.699255 17135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:44279 every 8 connection(s)
I20260925 05:07:47.700795 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:47.700974 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.684880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:47513 (local address 127.8.215.62:44293)
0925 05:07:47.685049 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.685054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.685072 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:07:47.685129 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.685137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.685217 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.685319 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:47.687201 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.687899 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.690385 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.690886 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.691026 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.694144 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:47.694151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:47.694157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:47.694237 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:47.697908 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.698920 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.698926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.698932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.699097 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.699578 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.699582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.699584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.699853 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:47.699959 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.700456 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.700610 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:07:47.701226 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.684163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.8.215.3:47513)
0925 05:07:47.684888 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.684908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.685025 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.685379 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.685389 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.686296 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:47.686999 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.687017 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.688049 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.688061 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.690234 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.690251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.691113 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.691124 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.691421 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.692298 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:47.692329 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:47.693862 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:47.698105 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.698109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.698113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.698763 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.699223 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:47.699235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:47.699238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:47.699456 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:47.699971 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:47.699975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:47.700205 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.700378 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.700691 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":1268,"threads_started":1}
I20260925 05:07:47.702848 17136 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44293
I20260925 05:07:47.703089 17136 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:47.703825 17136 heartbeater.cc:511] Master 127.8.215.62:44293 requested a full tablet report, sending...
I20260925 05:07:47.705235 16651 ts_manager.cc:194] Registered new tserver with Master: bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279)
I20260925 05:07:47.705981 16651 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:51829
I20260925 05:07:47.709079 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17001
I20260925 05:07:47.709277 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:47.709725 9052 external_mini_cluster.cc:1428] Setting key e2880c6c916cb98f2475d80093dde3ee
I20260925 05:07:47.712412 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:07:47.764290 16745 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:47.753802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.8.215.62:44293)
0925 05:07:47.754020 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.754026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.754261 (+ 235us) server_negotiation.cc:407] Connection header received
0925 05:07:47.754476 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.754481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.754540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.754675 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:47.755943 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.756691 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.758033 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.758459 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.758599 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.763981 (+ 5382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58774: BlockingRecv error: recv got EOF from 127.0.0.1:58774 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 05:07:47 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:47.781301 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.767826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.8.215.62:44293)
0925 05:07:47.768003 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.768007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.768332 (+ 325us) server_negotiation.cc:407] Connection header received
0925 05:07:47.768517 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.768522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.768625 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.768799 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:47.769864 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.770686 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.771866 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.772346 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.772536 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.776070 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:47.776075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:47.776078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:47.776111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:47.778558 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.779233 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.779236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.779239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.779350 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:47.779878 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:47.779883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:47.779887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:47.780174 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:47.780489 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.780773 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.780999 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:07:47.785176 16651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:47.792128 16651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:47.817054 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.803223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:44279 (local address 127.0.0.1:44284)
0925 05:07:47.804433 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.804480 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.804677 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.806776 (+ 2099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.806787 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.806828 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.807395 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.807411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.810534 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.810543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.813706 (+ 3163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.813727 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.816040 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.816045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.816265 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.816323 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.816426 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.816596 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":12149,"threads_started":1}
I20260925 05:07:47.817338 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.803573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.8.215.3:44279)
0925 05:07:47.805516 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.805528 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.805567 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:07:47.805678 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.805689 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.805991 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.807646 (+ 1655us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.808070 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.810344 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.813969 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.815836 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.816521 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.816794 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.816864 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.817021 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1730,"thread_start_us":1506,"threads_started":1}
I20260925 05:07:47.821048 17066 tablet_service.cc:1467] Processing CreateTablet for tablet 3be751b95b674ed680ab79dc1abfe68f (DEFAULT_TABLE table=test-table [id=3862a6c5d6ea40d0be85404c3beca584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:47.822477 17066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be751b95b674ed680ab79dc1abfe68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:47.824131 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.804200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.8.215.2:37501)
0925 05:07:47.806609 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.806619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.815491 (+ 8872us) server_negotiation.cc:407] Connection header received
0925 05:07:47.815803 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.815815 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.816098 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.816272 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.817298 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.818762 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.821791 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.823118 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.823268 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.823482 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.823701 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.823853 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2217,"thread_start_us":2102,"threads_started":1}
I20260925 05:07:47.824219 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.803600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:37501 (local address 127.0.0.1:55554)
0925 05:07:47.815354 (+ 11754us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.815377 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.815606 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.816510 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.816515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.816555 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.817111 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.817123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.818942 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.818946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.821607 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.821622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.823310 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.823317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.823476 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.823517 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.823645 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.823779 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11603,"mutex_wait_us":124,"spinlock_wait_cycles":2816,"thread_start_us":779,"threads_started":1}
I20260925 05:07:47.824798 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.805696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.8.215.1:46193)
0925 05:07:47.814975 (+ 9279us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.814988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.815888 (+ 900us) server_negotiation.cc:407] Connection header received
0925 05:07:47.815984 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.815990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.816167 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.816321 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.817231 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.818762 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.821766 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.823244 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.823360 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.823539 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.824321 (+ 782us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.824501 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9126,"thread_start_us":8987,"threads_started":1}
I20260925 05:07:47.825033 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.805314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46193 (local address 127.0.0.1:56988)
0925 05:07:47.815660 (+ 10346us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.815676 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.815773 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.816510 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.816515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.816549 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.817020 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.817038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.818937 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.818943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.821607 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.821622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.823681 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.823687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.823848 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.823879 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.823957 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.824089 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10237,"mutex_wait_us":201,"spinlock_wait_cycles":896,"thread_start_us":9740,"threads_started":1}
I20260925 05:07:47.827786 16927 tablet_service.cc:1467] Processing CreateTablet for tablet 3be751b95b674ed680ab79dc1abfe68f (DEFAULT_TABLE table=test-table [id=3862a6c5d6ea40d0be85404c3beca584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:47.828199 16788 tablet_service.cc:1467] Processing CreateTablet for tablet 3be751b95b674ed680ab79dc1abfe68f (DEFAULT_TABLE table=test-table [id=3862a6c5d6ea40d0be85404c3beca584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:47.829149 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be751b95b674ed680ab79dc1abfe68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:47.829356 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be751b95b674ed680ab79dc1abfe68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:47.839803 17160 tablet_bootstrap.cc:492] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Bootstrap starting.
I20260925 05:07:47.841982 17161 tablet_bootstrap.cc:492] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Bootstrap starting.
I20260925 05:07:47.843235 17162 tablet_bootstrap.cc:492] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Bootstrap starting.
I20260925 05:07:47.844203 17160 tablet_bootstrap.cc:654] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:47.845468 17160 log.cc:824] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:47.846166 17161 tablet_bootstrap.cc:654] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:47.846577 17162 tablet_bootstrap.cc:654] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:47.847553 17161 log.cc:824] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:47.847719 17162 log.cc:824] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:47.848858 17160 tablet_bootstrap.cc:492] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: No bootstrap required, opened a new log
I20260925 05:07:47.849154 17160 ts_tablet_manager.cc:1397] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 05:07:47.850618 17162 tablet_bootstrap.cc:492] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: No bootstrap required, opened a new log
I20260925 05:07:47.850680 17161 tablet_bootstrap.cc:492] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: No bootstrap required, opened a new log
I20260925 05:07:47.850828 17162 ts_tablet_manager.cc:1397] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:07:47.850899 17161 ts_tablet_manager.cc:1397] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 05:07:47.858850 17162 raft_consensus.cc:348] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.859295 17162 raft_consensus.cc:374] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:47.859457 17162 raft_consensus.cc:729] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef7e8c29c84993bbe95f50ad00ae97, State: Initialized, Role: FOLLOWER
I20260925 05:07:47.859371 17160 raft_consensus.cc:348] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.859771 17160 raft_consensus.cc:374] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:47.859865 17160 raft_consensus.cc:729] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Initialized, Role: FOLLOWER
I20260925 05:07:47.860394 17162 consensus_queue.cc:260] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.860519 17160 consensus_queue.cc:260] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.861066 17161 raft_consensus.cc:348] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.861543 17161 raft_consensus.cc:374] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:47.861663 17161 raft_consensus.cc:729] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b669f9f9a9154434b3af3cdc4ce12067, State: Initialized, Role: FOLLOWER
I20260925 05:07:47.862432 17161 consensus_queue.cc:260] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.863888 16997 heartbeater.cc:503] Master 127.8.215.62:44293 was elected leader, sending a full tablet report...
I20260925 05:07:47.864486 17161 ts_tablet_manager.cc:1428] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 05:07:47.865939 17136 heartbeater.cc:503] Master 127.8.215.62:44293 was elected leader, sending a full tablet report...
I20260925 05:07:47.866827 17160 ts_tablet_manager.cc:1428] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260925 05:07:47.869601 17162 ts_tablet_manager.cc:1428] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260925 05:07:47.885504 17168 raft_consensus.cc:482] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:47.885802 17168 raft_consensus.cc:504] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.887332 17168 leader_election.cc:288] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279), 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193)
I20260925 05:07:47.890650 17167 raft_consensus.cc:482] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:47.890945 17167 raft_consensus.cc:504] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.892874 17167 leader_election.cc:288] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501), 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193)
I20260925 05:07:47.896308 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.887860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:57341 (local address 127.8.215.3:44279)
0925 05:07:47.888010 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.888014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.888804 (+ 790us) server_negotiation.cc:407] Connection header received
0925 05:07:47.889022 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.889030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.889128 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.889222 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.889924 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.891034 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.893193 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.894770 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.894894 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.895044 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.895777 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.895950 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:07:47.897497 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.887790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:44279 (local address 127.8.215.2:57341)
0925 05:07:47.888665 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.888684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.888852 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.889238 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.889241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.889285 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.889748 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.889767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.891229 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.891233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.893006 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.893022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.895286 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.895295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.895544 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.895593 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.896563 (+ 970us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.896973 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":27,"thread_start_us":477,"threads_started":1}
I20260925 05:07:47.898693 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be751b95b674ed680ab79dc1abfe68f" candidate_uuid: "b669f9f9a9154434b3af3cdc4ce12067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc67742601a494db2223f374c1e7b05" is_pre_election: true
I20260925 05:07:47.899243 17090 raft_consensus.cc:2454] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b669f9f9a9154434b3af3cdc4ce12067 in term 0.
I20260925 05:07:47.900115 16879 leader_election.cc:302] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b669f9f9a9154434b3af3cdc4ce12067, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:47.901510 17168 raft_consensus.cc:2781] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:47.901678 17168 raft_consensus.cc:482] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:47.901791 17168 raft_consensus.cc:3037] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:47.902626 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.893212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:37501 (local address 127.8.215.3:40995)
0925 05:07:47.893411 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.893430 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.893541 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.893956 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.893961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.893987 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.894311 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.894333 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.896150 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.896158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.899349 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.899371 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.901846 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.901897 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.902130 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.902163 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.902252 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.902376 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:07:47.903015 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.887813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46193 (local address 127.8.215.2:52077)
0925 05:07:47.888898 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.888921 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.889028 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.889359 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.889364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.889401 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.889822 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.889834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.891178 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.891185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.893865 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.893876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.902146 (+ 8270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.902153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.902333 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.902364 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.902442 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.902574 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":73,"spinlock_wait_cycles":20352,"thread_start_us":470,"threads_started":1}
I20260925 05:07:47.903314 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.887675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:52077 (local address 127.8.215.1:46193)
0925 05:07:47.887871 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.887876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.889002 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 05:07:47.889143 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.889149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.889265 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.889350 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.889961 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.891033 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.900113 (+ 9080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.901327 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.901434 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.901537 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.902900 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.903060 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:07:47.903993 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.893308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:40995 (local address 127.8.215.2:37501)
0925 05:07:47.893471 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.893476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.893492 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:07:47.893681 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.893686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.893748 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.893825 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.894524 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.895929 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.899844 (+ 3915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.901675 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.901809 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.901909 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.903596 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.903732 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:07:47.904217 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be751b95b674ed680ab79dc1abfe68f" candidate_uuid: "b669f9f9a9154434b3af3cdc4ce12067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" is_pre_election: true
I20260925 05:07:47.904861 16813 raft_consensus.cc:2454] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b669f9f9a9154434b3af3cdc4ce12067 in term 0.
I20260925 05:07:47.905072 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be751b95b674ed680ab79dc1abfe68f" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067" is_pre_election: true
I20260925 05:07:47.906100 17168 raft_consensus.cc:504] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
I20260925 05:07:47.906524 16952 raft_consensus.cc:2379] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcc67742601a494db2223f374c1e7b05 in current term 1: Already voted for candidate b669f9f9a9154434b3af3cdc4ce12067 in this term.
I20260925 05:07:47.906919 17168 leader_election.cc:288] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [CANDIDATE]: Term 1 election: Requested vote from peers bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279), 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193)
I20260925 05:07:47.907085 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be751b95b674ed680ab79dc1abfe68f" candidate_uuid: "b669f9f9a9154434b3af3cdc4ce12067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc67742601a494db2223f374c1e7b05"
I20260925 05:07:47.907320 17090 raft_consensus.cc:3037] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:47.907734 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be751b95b674ed680ab79dc1abfe68f" candidate_uuid: "b669f9f9a9154434b3af3cdc4ce12067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
I20260925 05:07:47.907939 16813 raft_consensus.cc:3037] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:47.909588 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.893992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:47285 (local address 127.8.215.1:46193)
0925 05:07:47.902551 (+ 8559us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.902557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.902578 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:07:47.902630 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.902637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.902703 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.902794 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:47.903551 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.904863 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.907152 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.908528 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.908666 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.908806 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.909115 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.909258 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8438,"thread_start_us":9237,"threads_started":1}
I20260925 05:07:47.909621 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.893596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46193 (local address 127.8.215.3:47285)
0925 05:07:47.901142 (+ 7546us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:47.901156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:47.901291 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:47.903058 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:47.903063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:47.903086 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:47.903364 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.903374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.905030 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.905035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.906998 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:47.907011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.908759 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.908765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:47.908964 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.908995 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:07:47.909091 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:07:47.909228 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7441,"thread_start_us":7272,"threads_started":1}
I20260925 05:07:47.910205 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be751b95b674ed680ab79dc1abfe68f" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" is_pre_election: true
I20260925 05:07:47.910818 16813 raft_consensus.cc:2454] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b669f9f9a9154434b3af3cdc4ce12067 in term 1.
I20260925 05:07:47.910944 17090 raft_consensus.cc:2454] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b669f9f9a9154434b3af3cdc4ce12067 in term 1.
I20260925 05:07:47.910995 17020 leader_election.cc:302] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc67742601a494db2223f374c1e7b05; no voters: 86ef7e8c29c84993bbe95f50ad00ae97, b669f9f9a9154434b3af3cdc4ce12067
I20260925 05:07:47.911238 16881 leader_election.cc:302] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ef7e8c29c84993bbe95f50ad00ae97, b669f9f9a9154434b3af3cdc4ce12067; no voters:
I20260925 05:07:47.911578 17168 raft_consensus.cc:2781] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:47.911746 17167 raft_consensus.cc:2726] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 05:07:47.912051 17168 raft_consensus.cc:686] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 1 LEADER]: Becoming Leader. State: Replica: b669f9f9a9154434b3af3cdc4ce12067, State: Running, Role: LEADER
I20260925 05:07:47.912541 17168 consensus_queue.cc:237] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } }
W20260925 05:07:47.915323 16859 tablet.cc:2369] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:47.916319 16649 catalog_manager.cc:5495] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 reported cstate change: term changed from 0 to 1, leader changed from <none> to b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "b669f9f9a9154434b3af3cdc4ce12067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:47.932475 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:47.927187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.8.215.2:37501)
0925 05:07:47.927370 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:47.927376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:47.927474 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:07:47.927757 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:47.927761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:47.927848 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:47.927998 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:47.928381 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.929163 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.930473 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:47.930864 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:47.930991 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:47.931395 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:47.931900 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:47.932065 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:47.932115 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 05:07:47.932190 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:07:47.940814 16813 raft_consensus.cc:1261] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Refusing update from remote peer b669f9f9a9154434b3af3cdc4ce12067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:47.940887 17090 raft_consensus.cc:1261] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Refusing update from remote peer b669f9f9a9154434b3af3cdc4ce12067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:47.941680 17168 consensus_queue.cc:1035] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:47.942042 17173 consensus_queue.cc:1035] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:07:47.942206 16998 tablet.cc:2369] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:07:47.952788 17137 tablet.cc:2369] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:47.954088 17176 mvcc.cc:205] Tried to move back new op lower bound from 7333121507075125248 to 7333121506973278208. Current Snapshot: MvccSnapshot[applied={T|T < 7333121507075125248}]
I20260925 05:07:47.966883 16649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:47.967199 16649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:47.970774 16649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Sending DeleteTablet for 3 replicas of tablet 3be751b95b674ed680ab79dc1abfe68f
I20260925 05:07:47.971921 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be751b95b674ed680ab79dc1abfe68f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:47 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:56988
I20260925 05:07:47.972023 17066 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be751b95b674ed680ab79dc1abfe68f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:47 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:44284
I20260925 05:07:47.972644 16927 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be751b95b674ed680ab79dc1abfe68f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:47 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55554
I20260925 05:07:47.973042 16649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:07:47.973594 17188 tablet_replica.cc:330] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: stopping tablet replica
W20260925 05:07:47.973837 16649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:47.973984 17188 raft_consensus.cc:2229] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:47.974509 17188 raft_consensus.cc:2258] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:47.976099 17188 ts_tablet_manager.cc:1905] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:47.985946 16787 tablet_service.cc:1467] Processing CreateTablet for tablet 875e48cc443e458f81c1760f5b2c1af2 (DEFAULT_TABLE table=test-table [id=0fa3dcb9c52f44b8ab84e10e7ae00667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:47.986527 16787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875e48cc443e458f81c1760f5b2c1af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:47.986428 16927 tablet_service.cc:1467] Processing CreateTablet for tablet 875e48cc443e458f81c1760f5b2c1af2 (DEFAULT_TABLE table=test-table [id=0fa3dcb9c52f44b8ab84e10e7ae00667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:47.986954 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875e48cc443e458f81c1760f5b2c1af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:47.989308 17065 tablet_service.cc:1467] Processing CreateTablet for tablet 875e48cc443e458f81c1760f5b2c1af2 (DEFAULT_TABLE table=test-table [id=0fa3dcb9c52f44b8ab84e10e7ae00667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:47.989923 17065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875e48cc443e458f81c1760f5b2c1af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:47.993564 17162 tablet_bootstrap.cc:492] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Bootstrap starting.
I20260925 05:07:47.994092 17188 ts_tablet_manager.cc:1918] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:47.994310 17188 log.cc:1196] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/3be751b95b674ed680ab79dc1abfe68f
I20260925 05:07:47.994931 17188 ts_tablet_manager.cc:1939] T 3be751b95b674ed680ab79dc1abfe68f P b669f9f9a9154434b3af3cdc4ce12067: Deleting consensus metadata
I20260925 05:07:47.995100 17162 tablet_bootstrap.cc:654] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:47.996850 16638 catalog_manager.cc:4841] TS b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501): tablet 3be751b95b674ed680ab79dc1abfe68f (table test-table [id=3862a6c5d6ea40d0be85404c3beca584]) successfully deleted
I20260925 05:07:48.004534 17161 tablet_bootstrap.cc:492] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Bootstrap starting.
I20260925 05:07:48.006175 17161 tablet_bootstrap.cc:654] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.010144 17160 tablet_bootstrap.cc:492] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Bootstrap starting.
I20260925 05:07:48.012009 17160 tablet_bootstrap.cc:654] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.016700 17161 tablet_bootstrap.cc:492] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: No bootstrap required, opened a new log
I20260925 05:07:48.016882 17161 ts_tablet_manager.cc:1397] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260925 05:07:48.017722 17161 raft_consensus.cc:348] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.018150 17161 raft_consensus.cc:374] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.018245 17161 raft_consensus.cc:729] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b669f9f9a9154434b3af3cdc4ce12067, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.018496 17161 consensus_queue.cc:260] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.019716 17161 ts_tablet_manager.cc:1428] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 05:07:48.021713 17162 tablet_bootstrap.cc:492] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: No bootstrap required, opened a new log
I20260925 05:07:48.023329 17162 ts_tablet_manager.cc:1397] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.005s
I20260925 05:07:48.024785 17162 raft_consensus.cc:348] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.025043 17162 raft_consensus.cc:374] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.025164 17162 raft_consensus.cc:729] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef7e8c29c84993bbe95f50ad00ae97, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.025569 17162 consensus_queue.cc:260] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.026166 17186 tablet_replica.cc:330] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: stopping tablet replica
I20260925 05:07:48.026444 17160 tablet_bootstrap.cc:492] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: No bootstrap required, opened a new log
I20260925 05:07:48.026610 17160 ts_tablet_manager.cc:1397] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260925 05:07:48.026679 17186 raft_consensus.cc:2229] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:48.027307 17186 raft_consensus.cc:2258] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:48.027515 17160 raft_consensus.cc:348] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.027979 17160 raft_consensus.cc:374] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.028141 17187 tablet_replica.cc:330] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: stopping tablet replica
I20260925 05:07:48.028162 17160 raft_consensus.cc:729] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.028442 17162 ts_tablet_manager.cc:1428] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 05:07:48.028575 17187 raft_consensus.cc:2229] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:48.028954 17160 consensus_queue.cc:260] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.029404 17187 raft_consensus.cc:2258] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:48.030862 17160 ts_tablet_manager.cc:1428] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 05:07:48.031567 17186 ts_tablet_manager.cc:1905] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:48.033005 17167 raft_consensus.cc:482] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:48.033062 17187 ts_tablet_manager.cc:1905] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:48.033159 17167 raft_consensus.cc:504] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.033772 17167 leader_election.cc:288] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:48.034685 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875e48cc443e458f81c1760f5b2c1af2" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" is_pre_election: true
I20260925 05:07:48.034902 16813 raft_consensus.cc:2454] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:48.035293 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875e48cc443e458f81c1760f5b2c1af2" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067" is_pre_election: true
I20260925 05:07:48.035362 17020 leader_election.cc:302] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ef7e8c29c84993bbe95f50ad00ae97, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:48.035544 16952 raft_consensus.cc:2454] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:48.035874 17167 raft_consensus.cc:2781] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:48.035984 17167 raft_consensus.cc:482] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:48.036063 17167 raft_consensus.cc:3037] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.039171 17167 raft_consensus.cc:504] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.039757 17167 leader_election.cc:288] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Requested vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:48.040289 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875e48cc443e458f81c1760f5b2c1af2" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
I20260925 05:07:48.040460 17187 ts_tablet_manager.cc:1918] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:48.040520 16813 raft_consensus.cc:3037] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.040623 17187 log.cc:1196] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/3be751b95b674ed680ab79dc1abfe68f
I20260925 05:07:48.041194 17187 ts_tablet_manager.cc:1939] T 3be751b95b674ed680ab79dc1abfe68f P bcc67742601a494db2223f374c1e7b05: Deleting consensus metadata
I20260925 05:07:48.042933 16636 catalog_manager.cc:4841] TS bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279): tablet 3be751b95b674ed680ab79dc1abfe68f (table test-table [id=3862a6c5d6ea40d0be85404c3beca584]) successfully deleted
I20260925 05:07:48.043006 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875e48cc443e458f81c1760f5b2c1af2" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067"
I20260925 05:07:48.043267 16952 raft_consensus.cc:3037] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.043331 16813 raft_consensus.cc:2454] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:48.043816 17020 leader_election.cc:302] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ef7e8c29c84993bbe95f50ad00ae97, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:48.044111 17167 raft_consensus.cc:2781] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:48.044535 17167 raft_consensus.cc:686] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Becoming Leader. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Running, Role: LEADER
I20260925 05:07:48.044970 17167 consensus_queue.cc:237] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.046504 16952 raft_consensus.cc:2454] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:48.050961 16651 catalog_manager.cc:5495] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcc67742601a494db2223f374c1e7b05 (127.8.215.3). New cstate: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } health_report { overall_health: HEALTHY } } }
I20260925 05:07:48.052865 17186 ts_tablet_manager.cc:1918] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:48.053031 17186 log.cc:1196] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/3be751b95b674ed680ab79dc1abfe68f
I20260925 05:07:48.053637 17186 ts_tablet_manager.cc:1939] T 3be751b95b674ed680ab79dc1abfe68f P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting consensus metadata
I20260925 05:07:48.055326 16638 catalog_manager.cc:4841] TS 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193): tablet 3be751b95b674ed680ab79dc1abfe68f (table test-table [id=3862a6c5d6ea40d0be85404c3beca584]) successfully deleted
Sep 25 05:07:48 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312865, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.62@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:48.101660 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.087463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.8.215.62:44293)
0925 05:07:48.087710 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:48.087718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:48.088287 (+ 569us) server_negotiation.cc:407] Connection header received
0925 05:07:48.088439 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:48.088445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:48.088522 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:48.088638 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:48.089793 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.090970 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.092164 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.092806 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.092927 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.096135 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:48.096143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:48.096148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:48.096183 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:48.098979 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:48.099634 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:48.099639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:48.099643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:48.099765 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:48.100114 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:48.100121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:48.100125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:48.100372 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:48.100536 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:48.100960 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 05:07:48.101280 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":109}
I20260925 05:07:48.101693 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.087466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44293 (local address 127.0.0.1:58788)
0925 05:07:48.088184 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:48.088200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:48.088326 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:48.088670 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:48.088674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:48.089392 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:48.089652 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:48.089661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.091123 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.091126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:48.092037 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:48.092047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.092967 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.092971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:48.093078 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.093865 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:48.093882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:48.095944 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:48.099108 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:48.099122 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:48.099130 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:48.099521 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:48.099868 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:48.099874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:48.099878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:48.100011 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:48.100501 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:48.100509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:48.100680 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 05:07:48.100877 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 05:07:48.101163 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":412,"threads_started":1}
I20260925 05:07:48.106819 16651 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:58788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:07:48.107785 16651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:48.114400 16788 tablet_service.cc:1467] Processing CreateTablet for tablet df5db59b6421469f84c1c6940446fe25 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9aaa383644a74f6b90528fd8f598bbc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:07:48.114559 16927 tablet_service.cc:1467] Processing CreateTablet for tablet df5db59b6421469f84c1c6940446fe25 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9aaa383644a74f6b90528fd8f598bbc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:07:48.114980 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5db59b6421469f84c1c6940446fe25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:48.115109 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5db59b6421469f84c1c6940446fe25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:48.120165 17161 tablet_bootstrap.cc:492] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: Bootstrap starting.
I20260925 05:07:48.121837 17161 tablet_bootstrap.cc:654] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.121804 17066 tablet_service.cc:1467] Processing CreateTablet for tablet df5db59b6421469f84c1c6940446fe25 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9aaa383644a74f6b90528fd8f598bbc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:07:48.122330 17066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5db59b6421469f84c1c6940446fe25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:48.122915 17162 tablet_bootstrap.cc:492] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: Bootstrap starting.
I20260925 05:07:48.124490 17162 tablet_bootstrap.cc:654] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.131040 17160 tablet_bootstrap.cc:492] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: Bootstrap starting.
I20260925 05:07:48.132313 17160 tablet_bootstrap.cc:654] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.134414 17160 tablet_bootstrap.cc:492] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: No bootstrap required, opened a new log
I20260925 05:07:48.134593 17160 ts_tablet_manager.cc:1397] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 05:07:48.135401 17160 raft_consensus.cc:348] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.135661 17160 raft_consensus.cc:374] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.135747 17160 raft_consensus.cc:729] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.135907 17161 tablet_bootstrap.cc:492] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: No bootstrap required, opened a new log
I20260925 05:07:48.136126 17161 ts_tablet_manager.cc:1397] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20260925 05:07:48.136013 17160 consensus_queue.cc:260] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.136440 17167 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.137145 17161 raft_consensus.cc:348] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.137436 17161 raft_consensus.cc:374] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.137588 17161 raft_consensus.cc:729] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b669f9f9a9154434b3af3cdc4ce12067, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.137892 17161 consensus_queue.cc:260] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.138657 17161 ts_tablet_manager.cc:1428] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:07:48.136693 17167 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.138458 17173 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.139343 17173 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.139348 17160 ts_tablet_manager.cc:1428] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260925 05:07:48.142797 17162 tablet_bootstrap.cc:492] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: No bootstrap required, opened a new log
I20260925 05:07:48.142966 17162 ts_tablet_manager.cc:1397] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.005s
I20260925 05:07:48.143649 17162 raft_consensus.cc:348] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.143893 17162 raft_consensus.cc:374] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.143965 17162 raft_consensus.cc:729] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef7e8c29c84993bbe95f50ad00ae97, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.144207 17162 consensus_queue.cc:260] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.144820 17162 ts_tablet_manager.cc:1428] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:07:48.144709 17166 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.145124 17166 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.479074 17167 raft_consensus.cc:482] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:48.479306 17167 raft_consensus.cc:504] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.480023 17167 leader_election.cc:288] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:48.480638 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5db59b6421469f84c1c6940446fe25" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" is_pre_election: true
I20260925 05:07:48.480830 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5db59b6421469f84c1c6940446fe25" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067" is_pre_election: true
I20260925 05:07:48.481036 16813 raft_consensus.cc:2454] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:48.481122 16952 raft_consensus.cc:2454] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:48.481501 17020 leader_election.cc:302] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ef7e8c29c84993bbe95f50ad00ae97, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:48.481796 17167 raft_consensus.cc:2781] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:48.481884 17167 raft_consensus.cc:482] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:48.482013 17167 raft_consensus.cc:3037] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.483844 17167 raft_consensus.cc:504] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.484323 17167 leader_election.cc:288] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Requested vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:48.484874 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5db59b6421469f84c1c6940446fe25" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067"
I20260925 05:07:48.484876 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5db59b6421469f84c1c6940446fe25" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
I20260925 05:07:48.485075 16952 raft_consensus.cc:3037] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.485119 16813 raft_consensus.cc:3037] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.486927 16952 raft_consensus.cc:2454] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:48.487190 16813 raft_consensus.cc:2454] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:48.487341 17020 leader_election.cc:302] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b669f9f9a9154434b3af3cdc4ce12067, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:48.487624 17167 raft_consensus.cc:2781] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:48.487807 17167 raft_consensus.cc:686] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Becoming Leader. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Running, Role: LEADER
I20260925 05:07:48.488044 17167 consensus_queue.cc:237] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.489367 17192 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: TxnStatusTablet state changed. Reason: New leader bcc67742601a494db2223f374c1e7b05. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.489650 17192 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:07:48.490170 17204 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:07:48.490384 17204 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:07:48.491019 16649 catalog_manager.cc:5495] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcc67742601a494db2223f374c1e7b05 (127.8.215.3). New cstate: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } health_report { overall_health: HEALTHY } } }
I20260925 05:07:48.560696 17192 consensus_queue.cc:1035] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:48.567392 17192 consensus_queue.cc:1035] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:48.592470 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.586052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:44279 (local address 127.0.0.1:44294)
0925 05:07:48.586303 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:48.586329 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:48.586477 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:48.587134 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:48.587137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:48.587146 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:07:48.587301 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:48.587308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.588846 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.588850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:48.589820 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:48.589829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.590777 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.590781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:48.590919 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.590976 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:07:48.592059 (+ 1083us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:07:48.592077 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 05:07:48.592136 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:07:48.592244 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:07:48.592756 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.586144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.8.215.3:44279)
0925 05:07:48.586701 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:48.586707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:48.586736 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:07:48.586833 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:48.586838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:48.586939 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:48.587117 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:48.587438 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.588694 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.589959 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.590648 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.590790 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.591176 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:48.591924 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:48.592107 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:48.592156 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 05:07:48.592234 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":247,"threads_started":1}
I20260925 05:07:48.596154 16813 raft_consensus.cc:1261] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Refusing update from remote peer bcc67742601a494db2223f374c1e7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:48.596263 16952 raft_consensus.cc:1261] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Refusing update from remote peer bcc67742601a494db2223f374c1e7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:48.596836 17167 consensus_queue.cc:1035] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:48.597138 17192 consensus_queue.cc:1035] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:48.600381 17166 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: TxnStatusTablet state changed. Reason: New leader bcc67742601a494db2223f374c1e7b05. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.600625 17166 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.605659 17210 mvcc.cc:205] Tried to move back new op lower bound from 7333121509764153344 to 7333121509328887808. Current Snapshot: MvccSnapshot[applied={T|T < 7333121509764153344}]
I20260925 05:07:48.608243 17166 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.608477 17166 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P 86ef7e8c29c84993bbe95f50ad00ae97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.615391 17167 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.615713 17167 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:07:48.617800 17173 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: TxnStatusTablet state changed. Reason: New leader bcc67742601a494db2223f374c1e7b05. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.618060 17173 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.621205 17192 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.621479 17192 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P bcc67742601a494db2223f374c1e7b05: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:07:48.622129 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.615992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.8.215.3:44279)
0925 05:07:48.616188 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:48.616193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:48.616212 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:07:48.616273 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:48.616279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:48.616358 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:48.616453 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:48.617020 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.618160 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.620143 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.620627 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.620794 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.621258 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:48.621353 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:48.621568 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:48.621746 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 05:07:48.621828 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 05:07:48.626067 17173 tablet_replica.cc:439] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } } }
I20260925 05:07:48.626332 17173 tablet_replica.cc:442] T df5db59b6421469f84c1c6940446fe25 P b669f9f9a9154434b3af3cdc4ce12067: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:07:48.639219 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.633215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:33901 (local address 127.8.215.3:44279)
0925 05:07:48.633387 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:48.633391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:48.633962 (+ 571us) server_negotiation.cc:407] Connection header received
0925 05:07:48.634194 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:48.634200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:48.634279 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:48.634376 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:48.634945 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.635773 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.637360 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.637777 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.637939 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.638410 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:48.638498 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:48.638597 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:48.638701 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:07:48.638821 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":43}
I20260925 05:07:48.639271 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.633267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:44279 (local address 127.8.215.3:33901)
0925 05:07:48.633865 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:48.633881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:48.634054 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:48.634385 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:48.634389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:48.634399 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:07:48.634797 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:48.634810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.635947 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.635954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:48.637227 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:48.637239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.637991 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.637998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:48.638190 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.638239 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:07:48.638595 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:07:48.638610 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 05:07:48.638687 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:07:48.638857 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":17,"thread_start_us":347,"threads_started":1}
I20260925 05:07:48.727270 16651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:48.727551 16651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:48.731451 16651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Sending DeleteTablet for 3 replicas of tablet 875e48cc443e458f81c1760f5b2c1af2
I20260925 05:07:48.732211 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet 875e48cc443e458f81c1760f5b2c1af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:48 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:56988
I20260925 05:07:48.732968 17226 tablet_replica.cc:330] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: stopping tablet replica
I20260925 05:07:48.733305 17226 raft_consensus.cc:2229] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:48.733333 16927 tablet_service.cc:1514] Processing DeleteTablet for tablet 875e48cc443e458f81c1760f5b2c1af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:48 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55554
I20260925 05:07:48.733735 17226 raft_consensus.cc:2258] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:48.735234 17226 ts_tablet_manager.cc:1905] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:48.737010 17065 tablet_service.cc:1514] Processing DeleteTablet for tablet 875e48cc443e458f81c1760f5b2c1af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:48 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:44284
I20260925 05:07:48.741223 17227 tablet_replica.cc:330] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: stopping tablet replica
I20260925 05:07:48.742079 17227 raft_consensus.cc:2229] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:48.742416 17227 raft_consensus.cc:2258] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:48.743150 17228 tablet_replica.cc:330] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: stopping tablet replica
I20260925 05:07:48.743386 17228 raft_consensus.cc:2229] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:48.743989 17228 raft_consensus.cc:2258] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:48.744517 17227 ts_tablet_manager.cc:1905] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:48.746565 17226 ts_tablet_manager.cc:1918] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:48.749217 17228 ts_tablet_manager.cc:1905] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:48.749430 17226 log.cc:1196] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/875e48cc443e458f81c1760f5b2c1af2
I20260925 05:07:48.750029 17226 ts_tablet_manager.cc:1939] T 875e48cc443e458f81c1760f5b2c1af2 P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting consensus metadata
I20260925 05:07:48.753593 16638 catalog_manager.cc:4841] TS 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193): tablet 875e48cc443e458f81c1760f5b2c1af2 (table test-table [id=0fa3dcb9c52f44b8ab84e10e7ae00667]) successfully deleted
I20260925 05:07:48.756026 17227 ts_tablet_manager.cc:1918] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:48.756156 17227 log.cc:1196] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/875e48cc443e458f81c1760f5b2c1af2
I20260925 05:07:48.756579 17227 ts_tablet_manager.cc:1939] T 875e48cc443e458f81c1760f5b2c1af2 P b669f9f9a9154434b3af3cdc4ce12067: Deleting consensus metadata
I20260925 05:07:48.757656 16638 catalog_manager.cc:4841] TS b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501): tablet 875e48cc443e458f81c1760f5b2c1af2 (table test-table [id=0fa3dcb9c52f44b8ab84e10e7ae00667]) successfully deleted
I20260925 05:07:48.761478 17228 ts_tablet_manager.cc:1918] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:48.761693 17228 log.cc:1196] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/875e48cc443e458f81c1760f5b2c1af2
I20260925 05:07:48.762305 17228 ts_tablet_manager.cc:1939] T 875e48cc443e458f81c1760f5b2c1af2 P bcc67742601a494db2223f374c1e7b05: Deleting consensus metadata
I20260925 05:07:48.763582 16636 catalog_manager.cc:4841] TS bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279): tablet 875e48cc443e458f81c1760f5b2c1af2 (table test-table [id=0fa3dcb9c52f44b8ab84e10e7ae00667]) successfully deleted
Sep 25 05:07:48 dist-test-slave-wpdm krb5kdc[16606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:07:48 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:48 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:07:48.806210 17236 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:48.795410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.8.215.62:44293)
0925 05:07:48.796390 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:48.796394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:48.796409 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:07:48.796460 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:48.796464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:48.796517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:48.796641 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:48.797926 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.798751 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.800015 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.800435 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.800565 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.805840 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58792: BlockingRecv error: recv got EOF from 127.0.0.1:58792 (error 108)
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":723,"threads_started":1}
Sep 25 05:07:48 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM
I20260925 05:07:48.824509 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:48.809347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.8.215.62:44293)
0925 05:07:48.809563 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:48.809570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:48.809896 (+ 326us) server_negotiation.cc:407] Connection header received
0925 05:07:48.810081 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:48.810086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:48.810149 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:48.810232 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:48.811730 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.812557 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.814388 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:48.814841 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:48.814966 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:48.819343 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:48.819349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:48.819352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:48.819388 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:48.821945 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:48.822620 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:48.822630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:48.822632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:48.822759 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:48.823135 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:48.823141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:48.823144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:48.823409 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:48.823650 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:48.823945 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 05:07:48.824188 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:07:48.828277 16651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:48.828912 16651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:48.835978 16788 tablet_service.cc:1467] Processing CreateTablet for tablet 0ceb7ab208af4018a9d056c7f134970c (DEFAULT_TABLE table=test-table [id=c02391e5c82a4f7187664ebaee70b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:48.836123 16927 tablet_service.cc:1467] Processing CreateTablet for tablet 0ceb7ab208af4018a9d056c7f134970c (DEFAULT_TABLE table=test-table [id=c02391e5c82a4f7187664ebaee70b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:48.836411 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ceb7ab208af4018a9d056c7f134970c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:48.836565 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ceb7ab208af4018a9d056c7f134970c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:48.841368 17065 tablet_service.cc:1467] Processing CreateTablet for tablet 0ceb7ab208af4018a9d056c7f134970c (DEFAULT_TABLE table=test-table [id=c02391e5c82a4f7187664ebaee70b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:48.841780 17065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ceb7ab208af4018a9d056c7f134970c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:48.844003 17243 tablet_bootstrap.cc:492] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Bootstrap starting.
I20260925 05:07:48.845733 17243 tablet_bootstrap.cc:654] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.847832 17243 tablet_bootstrap.cc:492] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: No bootstrap required, opened a new log
I20260925 05:07:48.847985 17243 ts_tablet_manager.cc:1397] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 05:07:48.848932 17243 raft_consensus.cc:348] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.849165 17243 raft_consensus.cc:374] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.849270 17243 raft_consensus.cc:729] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b669f9f9a9154434b3af3cdc4ce12067, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.849454 17242 tablet_bootstrap.cc:492] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Bootstrap starting.
I20260925 05:07:48.849493 17243 consensus_queue.cc:260] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.852454 17242 tablet_bootstrap.cc:654] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.853250 17243 ts_tablet_manager.cc:1428] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 05:07:48.856500 17245 tablet_bootstrap.cc:492] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Bootstrap starting.
I20260925 05:07:48.858125 17242 tablet_bootstrap.cc:492] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: No bootstrap required, opened a new log
I20260925 05:07:48.858283 17242 ts_tablet_manager.cc:1397] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260925 05:07:48.858434 17245 tablet_bootstrap.cc:654] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:48.859077 17242 raft_consensus.cc:348] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.859340 17242 raft_consensus.cc:374] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.859438 17242 raft_consensus.cc:729] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef7e8c29c84993bbe95f50ad00ae97, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.859670 17242 consensus_queue.cc:260] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.860787 17242 ts_tablet_manager.cc:1428] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 05:07:48.862519 17245 tablet_bootstrap.cc:492] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: No bootstrap required, opened a new log
I20260925 05:07:48.862689 17245 ts_tablet_manager.cc:1397] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260925 05:07:48.863534 17245 raft_consensus.cc:348] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.863749 17245 raft_consensus.cc:374] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:48.863849 17245 raft_consensus.cc:729] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Initialized, Role: FOLLOWER
I20260925 05:07:48.864091 17245 consensus_queue.cc:260] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.864779 17245 ts_tablet_manager.cc:1428] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:07:48.952737 17192 raft_consensus.cc:482] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:48.952981 17192 raft_consensus.cc:504] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.953691 17192 leader_election.cc:288] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:48.954272 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceb7ab208af4018a9d056c7f134970c" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067" is_pre_election: true
I20260925 05:07:48.954273 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceb7ab208af4018a9d056c7f134970c" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" is_pre_election: true
I20260925 05:07:48.954550 16952 raft_consensus.cc:2454] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:48.954563 16812 raft_consensus.cc:2454] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:48.955070 17020 leader_election.cc:302] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b669f9f9a9154434b3af3cdc4ce12067, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:48.955343 17192 raft_consensus.cc:2781] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:48.955456 17192 raft_consensus.cc:482] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:48.955569 17192 raft_consensus.cc:3037] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.958895 17192 raft_consensus.cc:504] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.959617 17192 leader_election.cc:288] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Requested vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:48.960127 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceb7ab208af4018a9d056c7f134970c" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
I20260925 05:07:48.960222 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceb7ab208af4018a9d056c7f134970c" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067"
I20260925 05:07:48.960387 16812 raft_consensus.cc:3037] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.960484 16952 raft_consensus.cc:3037] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:48.962867 16952 raft_consensus.cc:2454] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:48.963383 17020 leader_election.cc:302] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b669f9f9a9154434b3af3cdc4ce12067, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:48.963428 16812 raft_consensus.cc:2454] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:48.963701 17192 raft_consensus.cc:2781] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:48.963896 17192 raft_consensus.cc:686] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Becoming Leader. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Running, Role: LEADER
I20260925 05:07:48.964161 17192 consensus_queue.cc:237] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:48.967084 16649 catalog_manager.cc:5495] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcc67742601a494db2223f374c1e7b05 (127.8.215.3). New cstate: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } health_report { overall_health: HEALTHY } } }
I20260925 05:07:49.010538 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:49.004914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.8.215.3:44279)
0925 05:07:49.005111 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:49.005116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:49.005216 (+ 100us) server_negotiation.cc:407] Connection header received
0925 05:07:49.005346 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:49.005351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:49.005460 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:49.005580 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:49.005890 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.007052 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.008572 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.009301 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.009424 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:49.009810 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:49.009930 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:49.010032 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:49.010186 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:07:49.010263 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:07:49.013298 16812 raft_consensus.cc:1261] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Refusing update from remote peer bcc67742601a494db2223f374c1e7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:49.013293 16952 raft_consensus.cc:1261] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Refusing update from remote peer bcc67742601a494db2223f374c1e7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:49.013978 17216 consensus_queue.cc:1035] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:49.014376 17192 consensus_queue.cc:1035] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:49.027943 16649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:49.028278 16649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:49.032168 16649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Sending DeleteTablet for 3 replicas of tablet 0ceb7ab208af4018a9d056c7f134970c
I20260925 05:07:49.033209 16927 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ceb7ab208af4018a9d056c7f134970c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55554
I20260925 05:07:49.033404 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ceb7ab208af4018a9d056c7f134970c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:56988
I20260925 05:07:49.033509 17227 tablet_replica.cc:330] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: stopping tablet replica
I20260925 05:07:49.033679 16787 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ceb7ab208af4018a9d056c7f134970c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:56988
I20260925 05:07:49.033700 17227 raft_consensus.cc:2229] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:49.033908 17227 raft_consensus.cc:2258] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:49.033900 17065 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ceb7ab208af4018a9d056c7f134970c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:44284
I20260925 05:07:49.034180 17226 tablet_replica.cc:330] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: stopping tablet replica
I20260925 05:07:49.034159 17228 tablet_replica.cc:330] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: stopping tablet replica
I20260925 05:07:49.034366 17226 raft_consensus.cc:2229] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:49.034373 17228 raft_consensus.cc:2229] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:49.034554 17226 raft_consensus.cc:2258] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:49.034711 17228 raft_consensus.cc:2258] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:49.035950 17227 ts_tablet_manager.cc:1905] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:49.036024 17228 ts_tablet_manager.cc:1905] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:49.036716 17226 ts_tablet_manager.cc:1905] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:49.040849 16649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:49.041640 16649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:49.042819 17227 ts_tablet_manager.cc:1918] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:49.042964 17227 log.cc:1196] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/0ceb7ab208af4018a9d056c7f134970c
I20260925 05:07:49.043361 17227 ts_tablet_manager.cc:1939] T 0ceb7ab208af4018a9d056c7f134970c P b669f9f9a9154434b3af3cdc4ce12067: Deleting consensus metadata
I20260925 05:07:49.043562 17226 ts_tablet_manager.cc:1918] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:49.043772 17226 log.cc:1196] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/0ceb7ab208af4018a9d056c7f134970c
I20260925 05:07:49.044417 16638 catalog_manager.cc:4841] TS b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501): tablet 0ceb7ab208af4018a9d056c7f134970c (table test-table [id=c02391e5c82a4f7187664ebaee70b832]) successfully deleted
I20260925 05:07:49.044251 17226 ts_tablet_manager.cc:1939] T 0ceb7ab208af4018a9d056c7f134970c P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting consensus metadata
I20260925 05:07:49.045674 16638 catalog_manager.cc:4841] TS 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193): tablet 0ceb7ab208af4018a9d056c7f134970c (table test-table [id=c02391e5c82a4f7187664ebaee70b832]) successfully deleted
W20260925 05:07:49.047317 16638 catalog_manager.cc:4808] TS 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193): delete failed for tablet 0ceb7ab208af4018a9d056c7f134970c because the tablet was not found. No further retry: Not found: Tablet not found: 0ceb7ab208af4018a9d056c7f134970c
I20260925 05:07:49.049479 17228 ts_tablet_manager.cc:1918] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:49.049412 16787 tablet_service.cc:1467] Processing CreateTablet for tablet d6600354244b4210bb928660bfda62f6 (DEFAULT_TABLE table=test-table [id=acff4f0e420a4dfab7802e247a380572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:49.049669 17228 log.cc:1196] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/0ceb7ab208af4018a9d056c7f134970c
I20260925 05:07:49.049823 16787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6600354244b4210bb928660bfda62f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:49.050102 17228 ts_tablet_manager.cc:1939] T 0ceb7ab208af4018a9d056c7f134970c P bcc67742601a494db2223f374c1e7b05: Deleting consensus metadata
I20260925 05:07:49.051069 16636 catalog_manager.cc:4841] TS bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279): tablet 0ceb7ab208af4018a9d056c7f134970c (table test-table [id=c02391e5c82a4f7187664ebaee70b832]) successfully deleted
I20260925 05:07:49.053320 16927 tablet_service.cc:1467] Processing CreateTablet for tablet d6600354244b4210bb928660bfda62f6 (DEFAULT_TABLE table=test-table [id=acff4f0e420a4dfab7802e247a380572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:49.053746 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6600354244b4210bb928660bfda62f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:49.053848 17065 tablet_service.cc:1467] Processing CreateTablet for tablet d6600354244b4210bb928660bfda62f6 (DEFAULT_TABLE table=test-table [id=acff4f0e420a4dfab7802e247a380572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:49.054273 17065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6600354244b4210bb928660bfda62f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:49.058774 17245 tablet_bootstrap.cc:492] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Bootstrap starting.
I20260925 05:07:49.060168 17243 tablet_bootstrap.cc:492] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Bootstrap starting.
I20260925 05:07:49.060340 17245 tablet_bootstrap.cc:654] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:49.061877 17243 tablet_bootstrap.cc:654] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:49.063978 17242 tablet_bootstrap.cc:492] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Bootstrap starting.
I20260925 05:07:49.065649 17242 tablet_bootstrap.cc:654] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:49.069126 17245 tablet_bootstrap.cc:492] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: No bootstrap required, opened a new log
I20260925 05:07:49.069553 17245 ts_tablet_manager.cc:1397] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260925 05:07:49.070631 17245 raft_consensus.cc:348] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.070962 17245 raft_consensus.cc:374] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:49.071115 17245 raft_consensus.cc:729] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Initialized, Role: FOLLOWER
I20260925 05:07:49.071460 17245 consensus_queue.cc:260] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.073673 17242 tablet_bootstrap.cc:492] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: No bootstrap required, opened a new log
I20260925 05:07:49.073839 17242 ts_tablet_manager.cc:1397] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260925 05:07:49.074254 17243 tablet_bootstrap.cc:492] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: No bootstrap required, opened a new log
I20260925 05:07:49.074421 17243 ts_tablet_manager.cc:1397] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20260925 05:07:49.074585 17242 raft_consensus.cc:348] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.074832 17242 raft_consensus.cc:374] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:49.074913 17242 raft_consensus.cc:729] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef7e8c29c84993bbe95f50ad00ae97, State: Initialized, Role: FOLLOWER
I20260925 05:07:49.075172 17245 ts_tablet_manager.cc:1428] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 05:07:49.075176 17242 consensus_queue.cc:260] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.075147 17243 raft_consensus.cc:348] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.075395 17243 raft_consensus.cc:374] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:49.075469 17243 raft_consensus.cc:729] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b669f9f9a9154434b3af3cdc4ce12067, State: Initialized, Role: FOLLOWER
I20260925 05:07:49.076705 17243 consensus_queue.cc:260] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.076869 17242 ts_tablet_manager.cc:1428] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:49.077813 17243 ts_tablet_manager.cc:1428] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 05:07:49.218987 17216 raft_consensus.cc:482] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:49.219210 17216 raft_consensus.cc:504] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.219874 17216 leader_election.cc:288] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:49.220433 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6600354244b4210bb928660bfda62f6" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" is_pre_election: true
I20260925 05:07:49.220439 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6600354244b4210bb928660bfda62f6" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067" is_pre_election: true
I20260925 05:07:49.220691 16812 raft_consensus.cc:2454] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:49.220698 16952 raft_consensus.cc:2454] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 0.
I20260925 05:07:49.221186 17020 leader_election.cc:302] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ef7e8c29c84993bbe95f50ad00ae97, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:49.221499 17216 raft_consensus.cc:2781] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:49.221604 17216 raft_consensus.cc:482] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:49.221724 17216 raft_consensus.cc:3037] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:49.223732 17216 raft_consensus.cc:504] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.224280 17216 leader_election.cc:288] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Requested vote from peers 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193), b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501)
I20260925 05:07:49.224812 16952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6600354244b4210bb928660bfda62f6" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b669f9f9a9154434b3af3cdc4ce12067"
I20260925 05:07:49.224854 16812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6600354244b4210bb928660bfda62f6" candidate_uuid: "bcc67742601a494db2223f374c1e7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef7e8c29c84993bbe95f50ad00ae97"
I20260925 05:07:49.225080 16952 raft_consensus.cc:3037] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:49.225087 16812 raft_consensus.cc:3037] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:49.228114 16952 raft_consensus.cc:2454] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:49.228117 16812 raft_consensus.cc:2454] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc67742601a494db2223f374c1e7b05 in term 1.
I20260925 05:07:49.228579 17020 leader_election.cc:302] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ef7e8c29c84993bbe95f50ad00ae97, bcc67742601a494db2223f374c1e7b05; no voters:
I20260925 05:07:49.228919 17216 raft_consensus.cc:2781] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:49.229297 17216 raft_consensus.cc:686] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Becoming Leader. State: Replica: bcc67742601a494db2223f374c1e7b05, State: Running, Role: LEADER
I20260925 05:07:49.229612 17216 consensus_queue.cc:237] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } }
I20260925 05:07:49.232317 16649 catalog_manager.cc:5495] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcc67742601a494db2223f374c1e7b05 (127.8.215.3). New cstate: current_term: 1 leader_uuid: "bcc67742601a494db2223f374c1e7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc67742601a494db2223f374c1e7b05" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44279 } health_report { overall_health: HEALTHY } } }
I20260925 05:07:49.271615 16952 raft_consensus.cc:1261] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Refusing update from remote peer bcc67742601a494db2223f374c1e7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:49.271999 16812 raft_consensus.cc:1261] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Refusing update from remote peer bcc67742601a494db2223f374c1e7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:49.272261 17192 consensus_queue.cc:1035] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b669f9f9a9154434b3af3cdc4ce12067" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:49.273124 17192 consensus_queue.cc:1035] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ef7e8c29c84993bbe95f50ad00ae97" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:49.278193 17207 mvcc.cc:205] Tried to move back new op lower bound from 7333121512532066304 to 7333121512366899200. Current Snapshot: MvccSnapshot[applied={T|T < 7333121512532066304}]
I20260925 05:07:49.326887 16650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:49.327184 16650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:49.330566 16650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaf7be440fb64604814d8621674c72d4: Sending DeleteTablet for 3 replicas of tablet d6600354244b4210bb928660bfda62f6
I20260925 05:07:49.331387 16787 tablet_service.cc:1514] Processing DeleteTablet for tablet d6600354244b4210bb928660bfda62f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:56988
I20260925 05:07:49.331692 17226 tablet_replica.cc:330] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: stopping tablet replica
I20260925 05:07:49.331683 16927 tablet_service.cc:1514] Processing DeleteTablet for tablet d6600354244b4210bb928660bfda62f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55554
I20260925 05:07:49.331913 17226 raft_consensus.cc:2229] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:49.332002 17227 tablet_replica.cc:330] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: stopping tablet replica
I20260925 05:07:49.332033 17066 tablet_service.cc:1514] Processing DeleteTablet for tablet d6600354244b4210bb928660bfda62f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:49 UTC) from {username='oryx', principal='oryx/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:44284
I20260925 05:07:49.332137 17226 raft_consensus.cc:2258] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:49.332190 17227 raft_consensus.cc:2229] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:49.332373 17227 raft_consensus.cc:2258] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:49.332429 17228 tablet_replica.cc:330] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: stopping tablet replica
I20260925 05:07:49.332641 17228 raft_consensus.cc:2229] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:49.333004 17228 raft_consensus.cc:2258] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:49.334350 17226 ts_tablet_manager.cc:1905] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:49.335049 17228 ts_tablet_manager.cc:1905] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:49.335451 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 16722
I20260925 05:07:49.336912 17227 ts_tablet_manager.cc:1905] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:49.342576 17228 ts_tablet_manager.cc:1918] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:49.342738 17228 log.cc:1196] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/d6600354244b4210bb928660bfda62f6
I20260925 05:07:49.343199 17228 ts_tablet_manager.cc:1939] T d6600354244b4210bb928660bfda62f6 P bcc67742601a494db2223f374c1e7b05: Deleting consensus metadata
I20260925 05:07:49.344435 16636 catalog_manager.cc:4841] TS bcc67742601a494db2223f374c1e7b05 (127.8.215.3:44279): tablet d6600354244b4210bb928660bfda62f6 (table test-table [id=acff4f0e420a4dfab7802e247a380572]) successfully deleted
I20260925 05:07:49.345425 17227 ts_tablet_manager.cc:1918] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:49.345582 17227 log.cc:1196] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/d6600354244b4210bb928660bfda62f6
I20260925 05:07:49.346081 17227 ts_tablet_manager.cc:1939] T d6600354244b4210bb928660bfda62f6 P b669f9f9a9154434b3af3cdc4ce12067: Deleting consensus metadata
I20260925 05:07:49.346390 17226 ts_tablet_manager.cc:1918] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:07:49.346575 17226 log.cc:1196] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/d6600354244b4210bb928660bfda62f6
I20260925 05:07:49.347064 17226 ts_tablet_manager.cc:1939] T d6600354244b4210bb928660bfda62f6 P 86ef7e8c29c84993bbe95f50ad00ae97: Deleting consensus metadata
I20260925 05:07:49.347290 16638 catalog_manager.cc:4841] TS b669f9f9a9154434b3af3cdc4ce12067 (127.8.215.2:37501): tablet d6600354244b4210bb928660bfda62f6 (table test-table [id=acff4f0e420a4dfab7802e247a380572]) successfully deleted
I20260925 05:07:49.349265 16638 catalog_manager.cc:4841] TS 86ef7e8c29c84993bbe95f50ad00ae97 (127.8.215.1:46193): tablet d6600354244b4210bb928660bfda62f6 (table test-table [id=acff4f0e420a4dfab7802e247a380572]) successfully deleted
Sep 25 05:07:49 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.1@KRBTEST.COM
I20260925 05:07:49.368074 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:49.340975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.8.215.1:46193)
0925 05:07:49.349246 (+ 8271us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:49.349253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:49.349843 (+ 590us) server_negotiation.cc:407] Connection header received
0925 05:07:49.350062 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:49.350065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:49.350134 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:49.350240 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:49.352037 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.352825 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.354807 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.355188 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.355277 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:49.360717 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:49.360779 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:49.360805 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:49.360841 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:49.364909 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:49.365689 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:49.365703 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:49.365710 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:49.365890 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:49.366284 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:49.366290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:49.366294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:49.366808 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:49.367156 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:49.367337 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 05:07:49.367645 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8101,"thread_start_us":7925,"threads_started":1}
I20260925 05:07:49.368863 16853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:49.464381 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 16722
I20260925 05:07:49.482795 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 16862
Sep 25 05:07:49 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.2@KRBTEST.COM
I20260925 05:07:49.497502 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:49.483581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.8.215.2:37501)
0925 05:07:49.483972 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:49.483976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:49.483992 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:07:49.484063 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:49.484067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:49.484127 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:49.484220 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:49.485691 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.486392 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.487557 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.488052 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.488212 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:49.492050 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:49.492063 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:49.492077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:49.492107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:49.494757 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:49.495372 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:49.495385 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:49.495388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:49.495511 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:49.495878 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:49.495884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:49.495887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:49.496219 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:49.496510 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:49.496979 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 05:07:49.497218 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":204,"threads_started":1}
I20260925 05:07:49.498119 16992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:49.617475 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 16862
I20260925 05:07:49.640048 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17001
Sep 25 05:07:49 dist-test-slave-wpdm krb5kdc[16606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.215.3@KRBTEST.COM
I20260925 05:07:49.654464 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:49.640792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.8.215.3:44279)
0925 05:07:49.641261 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:49.641266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:49.641282 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:07:49.641336 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:49.641340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:49.641395 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:49.641487 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:49.642841 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.643540 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.645085 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:49.645474 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:49.645570 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:49.649622 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:49.649637 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:49.649653 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:49.649687 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:49.652186 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:49.652831 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:49.652839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:49.652841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:49.652973 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:49.653284 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:49.653287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:49.653288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:49.653613 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:49.653826 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:49.654041 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 05:07:49.654208 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
I20260925 05:07:49.655066 17131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:49.794464 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17001
I20260925 05:07:49.820461 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 16619
I20260925 05:07:49.828428 16691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:49.978740 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 16619
2026-09-25T05:07:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5792 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:50 dist-test-slave-wpdm krb5kdc[17270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:50 dist-test-slave-wpdm krb5kdc[17270](info): set up 2 sockets
Sep 25 05:07:50 dist-test-slave-wpdm krb5kdc[17270](info): commencing operation
krb5kdc: starting...
W20260925 05:07:51.118289 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:51 dist-test-slave-wpdm krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:51.348361 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:37083
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:40031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:51.668129 17283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:51.668413 17283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:51.668499 17283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:51.675688 17283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:51.675789 17283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:51.675834 17283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:51.675890 17283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:51.675940 17283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:51.686264 17283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:40031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:37083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:51.687453 17283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:51.688694 17283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:51.700315 17288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:51.700265 17291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:51.700351 17289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:51.701082 17283 server_base.cc:1034] running on GCE node
I20260925 05:07:51.701792 17283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:51.703102 17283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:51.704332 17283 hybrid_clock.cc:648] HybridClock initialized: now 1790312871704292 us; error 48 us; skew 500 ppm
I20260925 05:07:51.704677 17283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T05:07:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1795 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:51 dist-test-slave-wpdm krb5kdc[17297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:51 dist-test-slave-wpdm krb5kdc[17297](info): set up 2 sockets
Sep 25 05:07:51 dist-test-slave-wpdm krb5kdc[17297](info): commencing operation
krb5kdc: starting...
W20260925 05:07:52.882197 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:07:53 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:07:53.118420 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40143
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:53.446585 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:53.446878 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:53.447065 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:53.454284 17310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:53.454394 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:53.454485 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:53.454566 17310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:53.454622 17310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:53.464357 17310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:53.465706 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:53.467046 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:53.479261 17315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.479147 17318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.479192 17316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.480600 17310 server_base.cc:1034] running on GCE node
I20260925 05:07:53.481448 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:53.482810 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:53.484025 17310 hybrid_clock.cc:648] HybridClock initialized: now 1790312873483968 us; error 36 us; skew 500 ppm
I20260925 05:07:53.484386 17310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:53 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.487437 17310 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:53.489014 17310 webserver.cc:460] Webserver started at http://127.8.215.62:33425/ using document root <none> and password file <none>
I20260925 05:07:53.489706 17310 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:53.489849 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:53.490188 17310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:53.493144 17310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "bd6caa27162d4a77ae663160ec84952d"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-wpdm"
server_key: "0973582cdb36e056c66beea06b03e702"
server_key_iv: "b741e69ff6fb8df3bb18618cfb00bb6a"
server_key_version: "encryptionkey@0"
I20260925 05:07:53.493914 17310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "bd6caa27162d4a77ae663160ec84952d"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-wpdm"
server_key: "0973582cdb36e056c66beea06b03e702"
server_key_iv: "b741e69ff6fb8df3bb18618cfb00bb6a"
server_key_version: "encryptionkey@0"
I20260925 05:07:53.499038 17310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 05:07:53.502452 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.503806 17310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:07:53.504009 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "bd6caa27162d4a77ae663160ec84952d"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-wpdm"
server_key: "0973582cdb36e056c66beea06b03e702"
server_key_iv: "b741e69ff6fb8df3bb18618cfb00bb6a"
server_key_version: "encryptionkey@0"
I20260925 05:07:53.504210 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:53.517747 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:53.521653 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:53.521945 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:53.540383 17310 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:40143
I20260925 05:07:53.540417 17376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:40143 every 8 connection(s)
I20260925 05:07:53.541930 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:53.545913 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:53.551385 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17310
I20260925 05:07:53.551600 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:07:53.552204 9052 external_mini_cluster.cc:1428] Setting key 23597206f11cca7cec41c48a4129cd28
I20260925 05:07:53.556881 17377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Bootstrap starting.
I20260925 05:07:53.560648 17377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:53.561954 17377 log.cc:824] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:53.565253 17377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: No bootstrap required, opened a new log
Sep 25 05:07:53 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:53.573369 17377 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:53.573778 17377 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:53.573920 17377 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6caa27162d4a77ae663160ec84952d, State: Initialized, Role: FOLLOWER
I20260925 05:07:53.574692 17377 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:53.574970 17377 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:53.575105 17377 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:53.575286 17377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:53.576363 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.554337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.8.215.62:40143)
0925 05:07:53.555176 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:53.555188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:53.555228 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:07:53.556928 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:53.556952 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:53.557389 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:53.557729 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:53.559582 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.560988 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.562809 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.563657 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.563797 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.568343 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:53.568351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:53.568367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:53.568407 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:53.571714 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.572470 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.572478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.572486 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.572652 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.573068 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.573074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.573077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.573646 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:53.573959 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:53.574427 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 05:07:53.574877 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":395,"threads_started":1}
I20260925 05:07:53.578483 17377 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:53.578990 17377 leader_election.cc:302] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd6caa27162d4a77ae663160ec84952d; no voters:
I20260925 05:07:53.579641 17377 leader_election.cc:288] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:53.579789 17382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:53.580463 17382 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 LEADER]: Becoming Leader. State: Replica: bd6caa27162d4a77ae663160ec84952d, State: Running, Role: LEADER
I20260925 05:07:53.581054 17382 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:53.581933 17377 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:53.584084 17383 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd6caa27162d4a77ae663160ec84952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } } }
I20260925 05:07:53.584508 17383 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:53.584074 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd6caa27162d4a77ae663160ec84952d. Latest consensus state: current_term: 1 leader_uuid: "bd6caa27162d4a77ae663160ec84952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } } }
I20260925 05:07:53.585012 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:53.585991 17389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:53.592314 17389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:53.609050 17389 catalog_manager.cc:1347] Generated new cluster ID: 43bd12342a6b4bf990fae710c8008989
I20260925 05:07:53.609169 17389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:53.639768 17389 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:53.641239 17389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:53.653178 17389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Generated new TSK 0
I20260925 05:07:53.654053 17389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:07:53.712208 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40143
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:54.050186 17403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:54.050486 17403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:54.050681 17403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:54.058075 17403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:54.058233 17403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:54.058410 17403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:54.068282 17403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:54.069661 17403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:54.071055 17403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:54.084830 17408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:54.084797 17409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:54.084932 17411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:54.086110 17403 server_base.cc:1034] running on GCE node
I20260925 05:07:54.086776 17403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:54.087730 17403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:54.088967 17403 hybrid_clock.cc:648] HybridClock initialized: now 1790312874088918 us; error 61 us; skew 500 ppm
I20260925 05:07:54.089309 17403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:54 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:54.092453 17403 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:07:54.094111 17403 webserver.cc:460] Webserver started at http://127.8.215.1:32785/ using document root <none> and password file <none>
I20260925 05:07:54.094805 17403 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:54.094947 17403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:54.095261 17403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:54.098416 17403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "aeeeba1bf8b949ebbfcbfd73e7af7d55"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "e10088f509ed7e79d2c7724598ba8737"
server_key_iv: "9c21aa946c0568c08f56e2e1062f8cf3"
server_key_version: "encryptionkey@0"
I20260925 05:07:54.099293 17403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "aeeeba1bf8b949ebbfcbfd73e7af7d55"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "e10088f509ed7e79d2c7724598ba8737"
server_key_iv: "9c21aa946c0568c08f56e2e1062f8cf3"
server_key_version: "encryptionkey@0"
I20260925 05:07:54.104565 17403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 05:07:54.108191 17418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.109725 17403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:54.109897 17403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "aeeeba1bf8b949ebbfcbfd73e7af7d55"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "e10088f509ed7e79d2c7724598ba8737"
server_key_iv: "9c21aa946c0568c08f56e2e1062f8cf3"
server_key_version: "encryptionkey@0"
I20260925 05:07:54.110107 17403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:54.126919 17403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:54.131409 17403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:54.131722 17403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:54.132663 17403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:54.133996 17403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:54.134069 17403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.134246 17403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:54.134310 17403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.165508 17403 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:45331
I20260925 05:07:54.165550 17530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:45331 every 8 connection(s)
I20260925 05:07:54.166821 17403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:54.175499 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17403
I20260925 05:07:54.175725 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:54.176350 9052 external_mini_cluster.cc:1428] Setting key cb2aa2df23c75453f8ed586fb290ad1d
Sep 25 05:07:54 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:54.186785 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.169353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:51111 (local address 127.8.215.62:40143)
0925 05:07:54.169760 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:54.169764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:54.170807 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 05:07:54.172649 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:54.172655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:54.172776 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:54.172904 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:54.174425 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.175144 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.176891 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.177457 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.177596 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.181137 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:54.181143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:54.181146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:54.181184 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:54.183770 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.184485 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.184492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.184501 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.184613 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.185061 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.185068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.185071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.185347 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:54.185501 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:54.186200 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 05:07:54.186415 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":206,"threads_started":1}
I20260925 05:07:54.187750 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.169944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:51111)
0925 05:07:54.170640 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:54.170692 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:54.172373 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:54.173152 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:54.173160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:54.173679 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:54.174256 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.174273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.175313 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.175317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.176711 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.176765 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.177691 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.177700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.177943 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.178750 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:54.178772 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:54.180933 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:54.183917 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.183925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.183943 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.184359 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.184756 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.184762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.184766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.184947 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.185474 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:54.185482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:54.185832 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 05:07:54.186096 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 05:07:54.186634 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
I20260925 05:07:54.189438 17531 heartbeater.cc:347] Connected to a master server at 127.8.215.62:40143
I20260925 05:07:54.189733 17531 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.190430 17531 heartbeater.cc:511] Master 127.8.215.62:40143 requested a full tablet report, sending...
I20260925 05:07:54.192718 17342 ts_manager.cc:194] Registered new tserver with Master: aeeeba1bf8b949ebbfcbfd73e7af7d55 (127.8.215.1:45331)
I20260925 05:07:54.194957 17342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:51111
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:07:54.228027 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40143
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:54.561014 17537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:54.561378 17537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:54.561511 17537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:54.570098 17537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:54.570223 17537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:54.570339 17537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:54.580385 17537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:54.581851 17537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:54.583443 17537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:54.597901 17542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:54.597901 17543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:54.598174 17545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:54.598999 17537 server_base.cc:1034] running on GCE node
I20260925 05:07:54.599711 17537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:54.600847 17537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:54.602093 17537 hybrid_clock.cc:648] HybridClock initialized: now 1790312874602044 us; error 46 us; skew 500 ppm
I20260925 05:07:54.602447 17537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:54 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:54.605880 17537 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:07:54.607638 17537 webserver.cc:460] Webserver started at http://127.8.215.2:41165/ using document root <none> and password file <none>
I20260925 05:07:54.608438 17537 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:54.608597 17537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:54.608985 17537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:54.612041 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "059823c3bdbe404a9ae9642c1c6b4863"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "8e60b92bbf3c6175cd6e2099cc2e1e52"
server_key_iv: "be229c2482f534732d367c23598ddfd6"
server_key_version: "encryptionkey@0"
I20260925 05:07:54.612983 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "059823c3bdbe404a9ae9642c1c6b4863"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "8e60b92bbf3c6175cd6e2099cc2e1e52"
server_key_iv: "be229c2482f534732d367c23598ddfd6"
server_key_version: "encryptionkey@0"
I20260925 05:07:54.618973 17537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:07:54.622886 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.624619 17537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:54.624895 17537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "059823c3bdbe404a9ae9642c1c6b4863"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "8e60b92bbf3c6175cd6e2099cc2e1e52"
server_key_iv: "be229c2482f534732d367c23598ddfd6"
server_key_version: "encryptionkey@0"
I20260925 05:07:54.625152 17537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:54.648538 17537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:54.653347 17537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:54.653679 17537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:54.654716 17537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:54.656127 17537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:54.656237 17537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.656397 17537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:54.656473 17537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.691447 17537 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:39473
I20260925 05:07:54.691494 17664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:39473 every 8 connection(s)
I20260925 05:07:54.692965 17537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:54.700673 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17537
I20260925 05:07:54.700927 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:54.701390 9052 external_mini_cluster.cc:1428] Setting key a44a930195164b5fe7440ab3e6043478
Sep 25 05:07:54 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:54.716380 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.695814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:53563 (local address 127.8.215.62:40143)
0925 05:07:54.696216 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:54.696221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:54.697385 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 05:07:54.699390 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:54.699395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:54.699465 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:54.699567 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:54.701666 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.702599 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.704632 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.705467 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.705617 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.709812 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:54.709817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:54.709820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:54.709853 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:54.712425 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.713507 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.713515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.713522 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.713706 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.714224 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.714233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.714239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.714618 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:54.714787 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:54.715641 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 05:07:54.715895 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":201,"threads_started":1}
I20260925 05:07:54.717069 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.696349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:53563)
0925 05:07:54.697202 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:54.697242 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:54.699057 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:54.699854 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:54.699864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:54.700509 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:54.701342 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.701363 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.702771 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.702777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.704391 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.704404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.705697 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.705702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.705911 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.706632 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:54.706681 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:54.709560 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:54.712634 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.712647 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.712672 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.713352 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.713872 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.713876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.713878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.714065 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.714813 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:54.714828 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:54.715225 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 05:07:54.715473 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 05:07:54.715890 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":377,"threads_started":1}
I20260925 05:07:54.718575 17665 heartbeater.cc:347] Connected to a master server at 127.8.215.62:40143
I20260925 05:07:54.718919 17665 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.719800 17665 heartbeater.cc:511] Master 127.8.215.62:40143 requested a full tablet report, sending...
I20260925 05:07:54.721438 17342 ts_manager.cc:194] Registered new tserver with Master: 059823c3bdbe404a9ae9642c1c6b4863 (127.8.215.2:39473)
I20260925 05:07:54.722286 17342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:53563
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:07:54.756582 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40143
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:55.112066 17671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:55.112385 17671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:55.112558 17671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:55.120456 17671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:55.120584 17671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:55.120842 17671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:55.131330 17671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:55.132781 17671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:55.134264 17671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:55.148005 17677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:55.148033 17679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:55.147979 17676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:55.148885 17671 server_base.cc:1034] running on GCE node
I20260925 05:07:55.149541 17671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:55.150547 17671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:55.151888 17671 hybrid_clock.cc:648] HybridClock initialized: now 1790312875151843 us; error 55 us; skew 500 ppm
I20260925 05:07:55.152275 17671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:55 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:55.155831 17671 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:07:55.157531 17671 webserver.cc:460] Webserver started at http://127.8.215.3:38591/ using document root <none> and password file <none>
I20260925 05:07:55.158277 17671 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:55.158434 17671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:55.158778 17671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:55.161958 17671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "9e1da2ee39a04c7baf93c94f829aed0b"
format_stamp: "Formatted at 2026-09-25 05:07:55 on dist-test-slave-wpdm"
server_key: "e6cc6a280adedc7972f644f423336823"
server_key_iv: "d9f0292e8f6c95efa75945749e12c47b"
server_key_version: "encryptionkey@0"
I20260925 05:07:55.162806 17671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "9e1da2ee39a04c7baf93c94f829aed0b"
format_stamp: "Formatted at 2026-09-25 05:07:55 on dist-test-slave-wpdm"
server_key: "e6cc6a280adedc7972f644f423336823"
server_key_iv: "d9f0292e8f6c95efa75945749e12c47b"
server_key_version: "encryptionkey@0"
I20260925 05:07:55.169301 17671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 05:07:55.173794 17686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:55.175205 17671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:07:55.175340 17671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "9e1da2ee39a04c7baf93c94f829aed0b"
format_stamp: "Formatted at 2026-09-25 05:07:55 on dist-test-slave-wpdm"
server_key: "e6cc6a280adedc7972f644f423336823"
server_key_iv: "d9f0292e8f6c95efa75945749e12c47b"
server_key_version: "encryptionkey@0"
I20260925 05:07:55.175513 17671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:55.189037 17671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:55.193444 17671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:55.193689 17671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:55.194685 17671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:55.195936 17671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:55.196007 17671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:55.196087 17671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:55.196133 17671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:55.198060 17531 heartbeater.cc:503] Master 127.8.215.62:40143 was elected leader, sending a full tablet report...
I20260925 05:07:55.230662 17671 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39449
I20260925 05:07:55.230721 17798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39449 every 8 connection(s)
I20260925 05:07:55.231959 17671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:55.240079 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17671
I20260925 05:07:55.240239 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:55.240679 9052 external_mini_cluster.cc:1428] Setting key cce6400220f4f65358dc6ede09194209
Sep 25 05:07:55 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:07:55.252136 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.234596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:49465 (local address 127.8.215.62:40143)
0925 05:07:55.235081 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.235086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.236138 (+ 1052us) server_negotiation.cc:407] Connection header received
0925 05:07:55.238078 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.238084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.238170 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.238288 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:55.239808 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.240694 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.242782 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.243202 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.243336 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.246703 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:55.246707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:55.246711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:55.246749 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:55.249438 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.250121 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.250124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.250127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.250238 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.250642 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.250645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.250647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.250899 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:55.251017 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.251718 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.251875 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
I20260925 05:07:55.253294 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.235239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:49465)
0925 05:07:55.235953 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.236005 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:55.237772 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:55.238521 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:55.238530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:55.239054 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:55.239617 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.239635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.240899 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.240905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.242645 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.242656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.243421 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.243428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.243659 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.244281 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:55.244302 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:55.246498 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:55.249583 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:55.249590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:55.249604 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:55.250012 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:55.250334 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:55.250337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:55.250340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:55.250515 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:55.251043 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:55.251049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:55.251367 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 05:07:55.251623 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 05:07:55.252019 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":290,"threads_started":1}
I20260925 05:07:55.254896 17799 heartbeater.cc:347] Connected to a master server at 127.8.215.62:40143
I20260925 05:07:55.255266 17799 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:55.256264 17799 heartbeater.cc:511] Master 127.8.215.62:40143 requested a full tablet report, sending...
I20260925 05:07:55.257807 17342 ts_manager.cc:194] Registered new tserver with Master: 9e1da2ee39a04c7baf93c94f829aed0b (127.8.215.3:39449)
I20260925 05:07:55.258591 17342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:49465
I20260925 05:07:55.268848 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab.
Entry for principal oryx/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab.
WARNING: no policy specified for oryx/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab.
Entry for principal oryx/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab.
WARNING: no policy specified for oryx/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab.
Entry for principal oryx/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab.
I20260925 05:07:55.419808 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17403
Sep 25 05:07:55 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:07:55.435799 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.421053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.8.215.1:45331)
0925 05:07:55.421769 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.421776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.421803 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:07:55.421862 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.421868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.422081 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.422231 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:55.423500 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.424506 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.425681 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.426243 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.426338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.430415 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:55.430426 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:55.430441 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:55.430475 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:55.433372 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.433973 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.433980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.433982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.434100 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.434451 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.434454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.434456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.434777 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:55.434989 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.435220 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.435412 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":405,"threads_started":1}
I20260925 05:07:55.437081 17526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:55.519829 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17403
I20260925 05:07:55.536623 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17537
Sep 25 05:07:55 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:07:55.553913 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.537522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.8.215.2:39473)
0925 05:07:55.538025 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.538033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.538058 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:07:55.538127 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.538134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.538355 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.538510 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:55.539669 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.540701 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.542574 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.543166 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.543275 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.548109 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:55.548125 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:55.548154 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:55.548199 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:55.551343 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.551968 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.551975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.551977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.552094 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.552417 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.552420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.552422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.552829 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:55.553046 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.553313 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.553504 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":258,"threads_started":1}
I20260925 05:07:55.555155 17660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:55.637698 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17537
I20260925 05:07:55.653563 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17671
Sep 25 05:07:55 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:07:55.669546 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.654511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.8.215.3:39449)
0925 05:07:55.655117 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.655124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.655151 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:07:55.655211 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.655218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.655430 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.655606 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:55.656961 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.657957 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.659240 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.659812 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.659973 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.664202 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:55.664213 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:55.664228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:55.664259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:55.666789 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.667449 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.667457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.667459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.667578 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.667946 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.667952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.667955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.668280 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:55.668595 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.668920 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.669177 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":321,"threads_started":1}
I20260925 05:07:55.671005 17794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:55.753870 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17671
I20260925 05:07:55.769609 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17310
I20260925 05:07:55.770836 17372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:07:55.872210 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17310
I20260925 05:07:55.893435 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40143
--webserver_interface=127.8.215.62
--webserver_port=33425
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:56.228849 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:56.229190 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:56.229377 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:56.236845 17817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:56.237001 17817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:56.237097 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:56.237208 17817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:56.237294 17817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:56.248263 17817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=33425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:56.249723 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:56.251073 17817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:56.262987 17825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:56.263036 17823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:56.262946 17822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:56.266235 17817 server_base.cc:1034] running on GCE node
I20260925 05:07:56.267102 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:56.268569 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:56.269809 17817 hybrid_clock.cc:648] HybridClock initialized: now 1790312876269777 us; error 44 us; skew 500 ppm
I20260925 05:07:56.270181 17817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:56.273386 17817 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:07:56.275033 17817 webserver.cc:460] Webserver started at http://127.8.215.62:33425/ using document root <none> and password file <none>
I20260925 05:07:56.275789 17817 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:56.275947 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:56.281807 17817 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:07:56.284652 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:56.286118 17817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:07:56.286305 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "bd6caa27162d4a77ae663160ec84952d"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-wpdm"
server_key: "0973582cdb36e056c66beea06b03e702"
server_key_iv: "b741e69ff6fb8df3bb18618cfb00bb6a"
server_key_version: "encryptionkey@0"
I20260925 05:07:56.287261 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:56.303177 17817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:56.307193 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:56.307471 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:56.326099 17817 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:40143
I20260925 05:07:56.326105 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:40143 every 8 connection(s)
I20260925 05:07:56.327612 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:56.334379 17884 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:07:56.336375 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17817
I20260925 05:07:56.337394 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:45331
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:40143
--webserver_port=32785
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:56.338467 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Bootstrap starting.
I20260925 05:07:56.359472 17884 log.cc:824] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:56.368049 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:07:56.368571 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Bootstrap complete.
I20260925 05:07:56.377861 17884 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:56.378665 17884 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6caa27162d4a77ae663160ec84952d, State: Initialized, Role: FOLLOWER
I20260925 05:07:56.379365 17884 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:56.379596 17884 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:56.379715 17884 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:56.379900 17884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:07:56.383584 17884 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:56.384106 17884 leader_election.cc:302] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd6caa27162d4a77ae663160ec84952d; no voters:
I20260925 05:07:56.384804 17884 leader_election.cc:288] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:07:56.384891 17888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:07:56.386219 17888 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [term 2 LEADER]: Becoming Leader. State: Replica: bd6caa27162d4a77ae663160ec84952d, State: Running, Role: LEADER
I20260925 05:07:56.386808 17888 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } }
I20260925 05:07:56.387101 17884 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:56.389528 17889 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bd6caa27162d4a77ae663160ec84952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } } }
I20260925 05:07:56.389945 17889 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:56.390348 17890 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd6caa27162d4a77ae663160ec84952d. Latest consensus state: current_term: 2 leader_uuid: "bd6caa27162d4a77ae663160ec84952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6caa27162d4a77ae663160ec84952d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 40143 } } }
I20260925 05:07:56.390811 17890 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:56.394878 17894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:56.400995 17894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:56.402395 17894 catalog_manager.cc:1259] Loaded cluster ID: 43bd12342a6b4bf990fae710c8008989
I20260925 05:07:56.402547 17894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:56.406450 17894 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:56.407945 17894 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bd6caa27162d4a77ae663160ec84952d: Loaded TSK: 0
I20260925 05:07:56.409451 17894 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:07:56.678087 17886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:56.678421 17886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:56.678594 17886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:56.686323 17886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:56.686475 17886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:56.686678 17886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:07:56.697217 17886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.1.keytab
--principal=oryx/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:45331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=32785
--tserver_master_addrs=127.8.215.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:56.698542 17886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:56.699851 17886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:56.713150 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:56.713171 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:56.713317 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:56.715544 17886 server_base.cc:1034] running on GCE node
I20260925 05:07:56.716127 17886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:56.717211 17886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:56.718503 17886 hybrid_clock.cc:648] HybridClock initialized: now 1790312876718397 us; error 85 us; skew 500 ppm
I20260925 05:07:56.718848 17886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312876, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:56.727007 17886 init.cc:377] Logged in from keytab as oryx/127.8.215.1@KRBTEST.COM (short username oryx)
I20260925 05:07:56.729066 17886 webserver.cc:460] Webserver started at http://127.8.215.1:32785/ using document root <none> and password file <none>
I20260925 05:07:56.729876 17886 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:56.730026 17886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:56.736428 17886 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:07:56.739543 17920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:56.741187 17886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:56.741382 17886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "aeeeba1bf8b949ebbfcbfd73e7af7d55"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "e10088f509ed7e79d2c7724598ba8737"
server_key_iv: "9c21aa946c0568c08f56e2e1062f8cf3"
server_key_version: "encryptionkey@0"
I20260925 05:07:56.742378 17886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:56.786486 17886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:56.791419 17886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:56.791699 17886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:56.792837 17886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:56.794163 17886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:56.794291 17886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:56.794416 17886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:56.794514 17886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:56.828950 17886 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:45331
I20260925 05:07:56.828990 18032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:45331 every 8 connection(s)
I20260925 05:07:56.830621 17886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:56.833832 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 17886
I20260925 05:07:56.834642 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:39473
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:40143
--webserver_port=41165
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:56.852083 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:56.833756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:34561)
0925 05:07:56.835121 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:56.835193 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:56.837913 (+ 2720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:56.839360 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:56.839373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:56.840181 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:56.841027 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:56.841055 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.842786 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.842793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:56.844110 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:56.844123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.845288 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.845296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:56.845618 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:56.846398 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:56.846423 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:56.850704 (+ 4281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":834,"threads_started":1}
W20260925 05:07:56.852375 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:56.833662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:34561 (local address 127.8.215.62:40143)
0925 05:07:56.834467 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:56.834486 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:56.835333 (+ 847us) server_negotiation.cc:407] Connection header received
0925 05:07:56.838505 (+ 3172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:56.838519 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:56.838858 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:56.839162 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:56.841261 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.842598 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.844276 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.845035 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.845260 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:56.851150 (+ 5890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:34561: BlockingRecv error: recv got EOF from 127.8.215.1:34561 (error 108)
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":312,"threads_started":1}
W20260925 05:07:56.853425 18033 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:40143 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.215.62:40143: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:56.862579 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:56.854120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:39311)
0925 05:07:56.854330 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:56.854351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:56.854483 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:56.854837 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:56.854842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:56.855194 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:56.855339 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:56.855347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.856543 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.856549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:56.857563 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:56.857575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.858345 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.858351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:56.858558 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:56.859186 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:56.859205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:56.862191 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:07:56.862620 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:56.854240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:39311 (local address 127.8.215.62:40143)
0925 05:07:56.854437 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:56.854443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:56.854464 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:07:56.854628 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:56.854633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:56.854704 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:56.854813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:56.855601 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.856383 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.857718 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.858185 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.858333 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:56.862387 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:39311: BlockingRecv error: recv got EOF from 127.8.215.1:39311 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:56 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:56.872237 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:56.863366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:48917)
0925 05:07:56.863568 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:56.863587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:56.863682 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:56.863979 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:56.863991 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:56.864362 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:56.864567 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:56.864579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.865681 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.865686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:56.867230 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:56.867244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.867947 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.867952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:56.868168 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:56.868789 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:56.868810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:56.871955 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:07:56.872484 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:56.863574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:48917 (local address 127.8.215.62:40143)
0925 05:07:56.863727 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:56.863731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:56.863747 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:07:56.863787 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:56.863790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:56.863856 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:56.863937 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:56.864788 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.865535 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.867388 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:56.867816 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:56.867909 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:56.872127 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:48917: BlockingRecv error: recv got EOF from 127.8.215.1:48917 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 05:07:56.872520 18033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:07:57.181077 18037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:57.181358 18037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:57.181449 18037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:57.188830 18037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:57.188937 18037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:57.189049 18037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:07:57.198972 18037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.2.keytab
--principal=oryx/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:39473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=41165
--tserver_master_addrs=127.8.215.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:57.200232 18037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:57.201536 18037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:57.214911 18042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.215011 18043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.214943 18045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.215796 18037 server_base.cc:1034] running on GCE node
I20260925 05:07:57.216454 18037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:57.217507 18037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:57.218742 18037 hybrid_clock.cc:648] HybridClock initialized: now 1790312877218689 us; error 49 us; skew 500 ppm
I20260925 05:07:57.219085 18037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:57.222396 18037 init.cc:377] Logged in from keytab as oryx/127.8.215.2@KRBTEST.COM (short username oryx)
I20260925 05:07:57.223974 18037 webserver.cc:460] Webserver started at http://127.8.215.2:41165/ using document root <none> and password file <none>
I20260925 05:07:57.224689 18037 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:57.224905 18037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:57.230443 18037 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:07:57.233299 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.234725 18037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:57.234913 18037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "059823c3bdbe404a9ae9642c1c6b4863"
format_stamp: "Formatted at 2026-09-25 05:07:54 on dist-test-slave-wpdm"
server_key: "8e60b92bbf3c6175cd6e2099cc2e1e52"
server_key_iv: "be229c2482f534732d367c23598ddfd6"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.235878 18037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:57.255080 18037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:57.259238 18037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:57.259498 18037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:57.260443 18037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:57.261780 18037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:57.261916 18037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.262040 18037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:57.262127 18037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.295822 18037 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:39473
I20260925 05:07:57.295879 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:39473 every 8 connection(s)
I20260925 05:07:57.297438 18037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:57.300004 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18037
I20260925 05:07:57.300549 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:39449
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:40143
--webserver_port=38591
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:07:57.314574 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.299975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:49703 (local address 127.8.215.62:40143)
0925 05:07:57.300194 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.300200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.301511 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 05:07:57.303317 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.303325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.303420 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.303545 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.305078 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.306237 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.308217 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.308618 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.308717 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.314333 (+ 5616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:49703: BlockingRecv error: recv got EOF from 127.8.215.2:49703 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:07:57.315294 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.300477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:49703)
0925 05:07:57.301324 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.301380 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.303029 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.303748 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.303757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.304278 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.304891 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.304910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.306409 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.306413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.308062 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.308078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.308867 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.308876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.309238 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.309952 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.309985 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.314122 (+ 4137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":375,"threads_started":1}
W20260925 05:07:57.316039 18165 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:40143 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.215.62:40143: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:57.326042 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.316670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:50941)
0925 05:07:57.316898 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.316919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.317031 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.317344 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.317351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.317786 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.318007 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.318022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.319152 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.319161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.320930 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.320948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.321616 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.321625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.321852 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.322432 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.322461 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.325769 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 05:07:57.326223 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.316832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50941 (local address 127.8.215.62:40143)
0925 05:07:57.317014 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.317021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.317039 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:07:57.317147 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.317150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.317206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.317281 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.318166 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.319015 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.321077 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.321459 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.321559 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.325954 (+ 4395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:50941: BlockingRecv error: recv got EOF from 127.8.215.2:50941 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:57.334543 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.326874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:35193)
0925 05:07:57.327090 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.327109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.327215 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.327528 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.327533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.327828 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.327962 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.327970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.329073 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.329080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.330059 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.330070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.330808 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.330814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.331002 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.331560 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.331583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.334251 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 05:07:57.334681 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.326932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:35193 (local address 127.8.215.62:40143)
0925 05:07:57.327102 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.327109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.327155 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:07:57.327339 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.327353 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.327406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.327510 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.328106 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.328945 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.330191 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.330664 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.330788 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.334431 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:35193: BlockingRecv error: recv got EOF from 127.8.215.2:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 05:07:57.335028 18165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:07:57.638345 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:57.638636 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:57.638734 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:57.646032 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:57.646147 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:57.646260 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:07:57.656117 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/krb5kdc/oryx_127.8.215.3.keytab
--principal=oryx/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:39449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=38591
--tserver_master_addrs=127.8.215.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:07:57.657434 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:57.658659 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:57.672803 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.672835 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.672876 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.673859 18168 server_base.cc:1034] running on GCE node
I20260925 05:07:57.674487 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:57.675449 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:57.676700 18168 hybrid_clock.cc:648] HybridClock initialized: now 1790312877676660 us; error 31 us; skew 500 ppm
I20260925 05:07:57.677110 18168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:57.680238 18168 init.cc:377] Logged in from keytab as oryx/127.8.215.3@KRBTEST.COM (short username oryx)
I20260925 05:07:57.681675 18168 webserver.cc:460] Webserver started at http://127.8.215.3:38591/ using document root <none> and password file <none>
I20260925 05:07:57.682345 18168 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:57.682510 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:57.687822 18168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 05:07:57.690587 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.691960 18168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:07:57.692132 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "9e1da2ee39a04c7baf93c94f829aed0b"
format_stamp: "Formatted at 2026-09-25 05:07:55 on dist-test-slave-wpdm"
server_key: "e6cc6a280adedc7972f644f423336823"
server_key_iv: "d9f0292e8f6c95efa75945749e12c47b"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.693154 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:57.714849 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:57.719120 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:57.719385 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:57.720484 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:57.721845 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:57.721961 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.722095 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:57.722190 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.755256 18168 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39449
I20260925 05:07:57.755352 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39449 every 8 connection(s)
I20260925 05:07:57.756862 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:57.765774 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18168
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:07:57.773953 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.759535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:49435 (local address 127.8.215.62:40143)
0925 05:07:57.759738 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.759742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.761024 (+ 1282us) server_negotiation.cc:407] Connection header received
0925 05:07:57.762815 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.762821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.762911 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.763019 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.764462 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.765227 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.767508 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.767993 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.768066 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.773700 (+ 5634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:49435: BlockingRecv error: recv got EOF from 127.8.215.3:49435 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:07:57.774915 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.760090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:49435)
0925 05:07:57.760833 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.760871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.762543 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.763263 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.763271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.763754 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.764292 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.764308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.765434 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.765440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.767297 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.767317 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.768327 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.768331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.768516 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.769306 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.769332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.773508 (+ 4176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":283,"threads_started":1}
W20260925 05:07:57.775640 18296 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:40143 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.215.62:40143: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:57.783551 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.766869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.8.215.62:40143)
0925 05:07:57.767775 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.767780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.767798 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:07:57.767840 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.767844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.767898 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.768036 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.769919 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.771055 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.772379 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.773158 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.773303 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.776575 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.776589 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.776611 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.776672 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.780112 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.780879 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.780893 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.780903 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.781173 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.781607 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.781614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.781618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.782148 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.782495 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.782848 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.783146 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":633,"threads_started":1}
I20260925 05:07:57.784540 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.776175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:42851)
0925 05:07:57.776598 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.776619 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.776757 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.777080 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.777085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.777561 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.777783 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.777799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.778849 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.778857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.779742 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.779752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.780459 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.780466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.780636 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.781200 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.781216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.784257 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271}
W20260925 05:07:57.784691 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.776398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:42851 (local address 127.8.215.62:40143)
0925 05:07:57.776622 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.776625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.776688 (+ 63us) server_negotiation.cc:407] Connection header received
0925 05:07:57.776873 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.776876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.776931 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.777018 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.777947 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.778704 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.779887 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.780318 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.780448 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.784449 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:42851: BlockingRecv error: recv got EOF from 127.8.215.3:42851 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:57.792892 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.785303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:46613)
0925 05:07:57.785456 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.785471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.785573 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.785911 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.785915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.786300 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.786495 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.786507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.787618 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.787628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.788586 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.788597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.789306 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.789311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.789494 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.790042 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.790059 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.792611 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 05:07:57.793084 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.785395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:46613 (local address 127.8.215.62:40143)
0925 05:07:57.785677 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.785682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.785708 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:07:57.785742 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.785746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.785790 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.785887 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.786641 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.787481 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.788765 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.789164 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.789297 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.792860 (+ 3563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:46613: BlockingRecv error: recv got EOF from 127.8.215.3:46613 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260925 05:07:57.793176 18296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:57 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:57.881625 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.873109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:50269)
0925 05:07:57.873631 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.873650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.873798 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.874138 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.874145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.874638 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.874821 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.874829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.875830 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.875837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.876870 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.876882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.877622 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.877628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.877799 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.878445 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.878462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.881301 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":279,"threads_started":1}
W20260925 05:07:57.881726 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:57.873194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:50269 (local address 127.8.215.62:40143)
0925 05:07:57.873438 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.873446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.873729 (+ 283us) server_negotiation.cc:407] Connection header received
0925 05:07:57.873931 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.873935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.873994 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.874098 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.874959 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.875684 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.877016 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.877443 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.877596 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.881484 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:50269: BlockingRecv error: recv got EOF from 127.8.215.1:50269 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 05:07:58 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:58 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:58.344604 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:58.335638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:51667)
0925 05:07:58.336144 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:58.336164 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:58.336275 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:58.336623 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:58.336626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:58.337322 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:58.337525 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:58.337535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.339010 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.339015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:58.340040 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:58.340058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.340814 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.340820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:58.340945 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:58.341511 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:58.341527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:58.344357 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
W20260925 05:07:58.344957 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:58.335760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:51667 (local address 127.8.215.62:40143)
0925 05:07:58.335933 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:58.335937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:58.336236 (+ 299us) server_negotiation.cc:407] Connection header received
0925 05:07:58.336420 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:58.336424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:58.336494 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:58.336584 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:58.337712 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.338805 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.340188 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.340616 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.340709 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:58.344544 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:51667: BlockingRecv error: recv got EOF from 127.8.215.2:51667 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 05:07:58 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:58 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:58.802682 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:58.793733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:40775)
0925 05:07:58.794239 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:58.794254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:58.794407 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:58.794739 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:58.794743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:58.795207 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:58.795388 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:58.795398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.796485 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.796490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:58.797603 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:58.797615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.798404 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.798410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:58.798585 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:58.799146 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:58.799161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:58.802358 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":248,"threads_started":1}
W20260925 05:07:58.802752 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:58.793772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:40775 (local address 127.8.215.62:40143)
0925 05:07:58.793999 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:58.794007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:58.794342 (+ 335us) server_negotiation.cc:407] Connection header received
0925 05:07:58.794528 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:58.794540 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:58.794601 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:58.794701 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:58.795534 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.796335 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.797760 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.798226 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.798366 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:58.802541 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:40775: BlockingRecv error: recv got EOF from 127.8.215.3:40775 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 05:07:58 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:58 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:58.891599 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:58.882704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:42729)
0925 05:07:58.883180 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:58.883199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:58.883311 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:58.883645 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:58.883649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:58.884065 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:58.884256 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:58.884267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.885638 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.885642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:58.886535 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:58.886546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.887207 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.887212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:58.887408 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:58.888090 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:58.888107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:58.891227 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
W20260925 05:07:58.891664 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:58.882786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:42729 (local address 127.8.215.62:40143)
0925 05:07:58.883026 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:58.883032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:58.883293 (+ 261us) server_negotiation.cc:407] Connection header received
0925 05:07:58.883412 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:58.883415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:58.883476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:58.883590 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:58.884435 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.885505 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.886661 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:58.887043 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:58.887140 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:58.891416 (+ 4276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:42729: BlockingRecv error: recv got EOF from 127.8.215.1:42729 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 05:07:59 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:59 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:59.354427 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:59.345538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:48591)
0925 05:07:59.346085 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:59.346115 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:59.346286 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:59.346630 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:59.346637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:59.347030 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:59.347216 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:59.347233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.348229 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.348236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:59.349168 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:59.349178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.349903 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.349909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:59.350072 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:59.350947 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:59.350964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:59.354058 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":298,"threads_started":1}
W20260925 05:07:59.354497 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:59.345615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:48591 (local address 127.8.215.62:40143)
0925 05:07:59.345882 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:59.345889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:59.346192 (+ 303us) server_negotiation.cc:407] Connection header received
0925 05:07:59.346408 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:59.346414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:59.346487 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:59.346590 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:59.347358 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.348095 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.349301 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.349754 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.349888 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:59.354279 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:48591: BlockingRecv error: recv got EOF from 127.8.215.2:48591 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 25 05:07:59 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:59 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:59.811880 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:59.803569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:51575)
0925 05:07:59.804110 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:59.804127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:59.804288 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:59.804651 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:59.804656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:59.805195 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:59.805334 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:59.805343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.806703 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.806707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:59.807600 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:59.807611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.808254 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.808257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:59.808361 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:59.809001 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:59.809023 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:59.811656 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":262,"threads_started":1}
W20260925 05:07:59.812053 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:59.803813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:51575 (local address 127.8.215.62:40143)
0925 05:07:59.804046 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:59.804052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:59.804216 (+ 164us) server_negotiation.cc:407] Connection header received
0925 05:07:59.804413 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:59.804419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:59.804507 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:59.804612 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:59.805530 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.806585 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.807726 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.808129 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.808227 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:59.811823 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:51575: BlockingRecv error: recv got EOF from 127.8.215.3:51575 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 05:07:59 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:07:59 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:07:59.900574 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:59.892477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:52591)
0925 05:07:59.893036 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:59.893057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:59.893174 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:59.893434 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:59.893437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:59.893743 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:59.893896 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:59.893903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.894856 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.894859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:59.895793 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:59.895804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.896439 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.896445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:59.896618 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:59.897342 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:59.897358 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:59.900236 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":394,"threads_started":1}
W20260925 05:07:59.900691 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:07:59.892547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:52591 (local address 127.8.215.62:40143)
0925 05:07:59.892821 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:59.892830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:59.893144 (+ 314us) server_negotiation.cc:407] Connection header received
0925 05:07:59.893262 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:59.893266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:59.893323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:59.893398 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:59.894006 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.894745 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.895901 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:59.896279 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:59.896427 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:59.900457 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:52591: BlockingRecv error: recv got EOF from 127.8.215.1:52591 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 05:08:00 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:00 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:00.363121 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:00.355398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:44927)
0925 05:08:00.355835 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:00.355855 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:00.355964 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:00.356228 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:00.356231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:00.356542 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:00.356698 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:00.356705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.357713 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.357717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:00.358609 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:00.358620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.359221 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.359225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:00.359340 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.359967 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:00.359983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:00.362880 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":255,"threads_started":1}
W20260925 05:08:00.363278 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:00.355551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:44927 (local address 127.8.215.62:40143)
0925 05:08:00.355794 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:00.355800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:00.355933 (+ 133us) server_negotiation.cc:407] Connection header received
0925 05:08:00.356045 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:00.356048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:00.356105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:00.356182 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:00.356888 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.357601 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.358730 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.359109 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.359224 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.363049 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:44927: BlockingRecv error: recv got EOF from 127.8.215.2:44927 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 05:08:00 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:00 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:00.821460 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:00.812975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:51301)
0925 05:08:00.813423 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:00.813439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:00.813549 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:00.813965 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:00.813970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:00.814348 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:00.814558 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:00.814570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.815903 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.815906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:00.816880 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:00.816891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.817571 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.817576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:00.817756 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.818433 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:00.818459 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:00.821104 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":265,"threads_started":1}
W20260925 05:08:00.821535 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:00.813194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:51301 (local address 127.8.215.62:40143)
0925 05:08:00.813402 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:00.813411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:00.813518 (+ 107us) server_negotiation.cc:407] Connection header received
0925 05:08:00.813695 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:00.813701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:00.813792 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:00.813902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:00.814757 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.815796 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.817003 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.817421 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.817545 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.821310 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:51301: BlockingRecv error: recv got EOF from 127.8.215.3:51301 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 05:08:00 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:00 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:00.909894 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:00.901481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:44147)
0925 05:08:00.901993 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:00.902011 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:00.902179 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:00.902501 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:00.902506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:00.902904 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:00.903050 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:00.903058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.904054 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.904061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:00.905089 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:00.905100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.905795 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.905801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:00.905980 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.906558 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:00.906574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:00.909554 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
W20260925 05:08:00.910006 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:00.901525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:44147 (local address 127.8.215.62:40143)
0925 05:08:00.901752 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:00.901761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:00.902098 (+ 337us) server_negotiation.cc:407] Connection header received
0925 05:08:00.902299 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:00.902304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:00.902370 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:00.902470 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:00.903198 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.903916 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.905222 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.905650 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.905774 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.909737 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:44147: BlockingRecv error: recv got EOF from 127.8.215.1:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 05:08:01 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:01 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:01.372359 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.364246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:60957)
0925 05:08:01.364807 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:01.364831 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:01.364979 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:01.365379 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:01.365384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:01.365881 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:01.366067 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.366086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.367206 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.367209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.368140 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.368150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.368844 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.368848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.368979 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.369517 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:01.369533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:01.372138 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":270,"threads_started":1}
W20260925 05:08:01.372535 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:01.364344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:60957 (local address 127.8.215.62:40143)
0925 05:08:01.364612 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:01.364618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:01.364923 (+ 305us) server_negotiation.cc:407] Connection header received
0925 05:08:01.365117 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:01.365125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:01.365229 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:01.365314 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:01.366238 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.367095 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.368267 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.368662 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.368845 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.372328 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:60957: BlockingRecv error: recv got EOF from 127.8.215.2:60957 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 25 05:08:01 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:01 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:01.842635 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.822370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:55789)
0925 05:08:01.822885 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:01.822900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:01.823056 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:01.823411 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:01.823416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:01.823798 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:01.823954 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.823962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.825462 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.825466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.826411 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.826421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.827064 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.827068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.827235 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.839530 (+ 12295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:01.839551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:01.842409 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":302,"threads_started":1}
W20260925 05:08:01.842846 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:01.822455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:55789 (local address 127.8.215.62:40143)
0925 05:08:01.822735 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:01.822742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:01.822983 (+ 241us) server_negotiation.cc:407] Connection header received
0925 05:08:01.823220 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:01.823224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:01.823280 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:01.823382 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:01.824121 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.825333 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.826530 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.826949 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.827040 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.842607 (+ 15567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:55789: BlockingRecv error: recv got EOF from 127.8.215.3:55789 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 05:08:01 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:01 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:01.918728 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.910867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:59933)
0925 05:08:01.911357 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:01.911385 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:01.911492 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:01.911768 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:01.911771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:01.912149 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:01.912307 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.912315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.913399 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.913403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.914337 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.914348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.915025 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.915029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.915143 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.915695 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:01.915711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:01.918407 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
W20260925 05:08:01.918864 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:01.910950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:59933 (local address 127.8.215.62:40143)
0925 05:08:01.911115 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:01.911119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:01.911488 (+ 369us) server_negotiation.cc:407] Connection header received
0925 05:08:01.911576 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:01.911579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:01.911640 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:01.911723 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:01.912450 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.913279 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.914470 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.914905 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.915000 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.918639 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:59933: BlockingRecv error: recv got EOF from 127.8.215.1:59933 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 05:08:02 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:02 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:02.381785 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.373417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:58615)
0925 05:08:02.373937 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.373957 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.374063 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.374356 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.374360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.374744 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.374901 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.374909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.375876 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.375880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.377041 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.377058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.378011 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.378017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.378143 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.378740 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.378757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.381547 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":270,"threads_started":1}
W20260925 05:08:02.381953 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:02.373678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:58615 (local address 127.8.215.62:40143)
0925 05:08:02.373852 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.373856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.374029 (+ 173us) server_negotiation.cc:407] Connection header received
0925 05:08:02.374154 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.374157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.374215 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.374306 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.375030 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.375764 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.377194 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.377860 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.377979 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.381732 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:58615: BlockingRecv error: recv got EOF from 127.8.215.2:58615 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 05:08:02 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:02 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:02.851557 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.843659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:59423)
0925 05:08:02.844172 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.844187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.844294 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.844564 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.844567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.844954 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.845126 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.845135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.846107 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.846111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.847049 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.847060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.847794 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.847798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.847913 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.848564 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.848580 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.851319 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":247,"threads_started":1}
W20260925 05:08:02.851678 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:02.843745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:59423 (local address 127.8.215.62:40143)
0925 05:08:02.843965 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.843972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.844266 (+ 294us) server_negotiation.cc:407] Connection header received
0925 05:08:02.844376 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.844380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.844442 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.844518 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.845254 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.845961 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.847198 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.847673 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.847768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.851483 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:59423: BlockingRecv error: recv got EOF from 127.8.215.3:59423 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 05:08:02 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:02 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:02.927860 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.919625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:33029)
0925 05:08:02.920124 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.920144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.920318 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.920636 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.920640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.921062 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.921203 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.921213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.922197 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.922202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.923109 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.923119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.923815 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.923821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.923995 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.924662 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.924683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.927648 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
W20260925 05:08:02.928000 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:02.919717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:33029 (local address 127.8.215.62:40143)
0925 05:08:02.919973 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.919981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.920251 (+ 270us) server_negotiation.cc:407] Connection header received
0925 05:08:02.920440 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.920451 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.920508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.920610 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.921343 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.922058 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.923257 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.923661 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.923792 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.927794 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:33029: BlockingRecv error: recv got EOF from 127.8.215.1:33029 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 05:08:03 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:03 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:03.391435 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.382850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:38647)
0925 05:08:03.383725 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.383755 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:03.383913 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:03.384325 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:03.384329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:03.384706 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:03.384920 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:03.384928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.385910 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.385914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:03.386838 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:03.386849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.387642 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.387646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:03.387769 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:03.388417 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:03.388434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:03.391102 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":574,"threads_started":1}
W20260925 05:08:03.391615 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:03.383055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:38647 (local address 127.8.215.62:40143)
0925 05:08:03.383343 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:03.383351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:03.383859 (+ 508us) server_negotiation.cc:407] Connection header received
0925 05:08:03.384074 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:03.384079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:03.384186 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:03.384280 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:03.385060 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.385789 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.386968 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.387415 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.387508 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:03.391370 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:38647: BlockingRecv error: recv got EOF from 127.8.215.2:38647 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 05:08:03 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:03 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:03.860328 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.852525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:56107)
0925 05:08:03.853151 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.853172 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:03.853307 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:03.853702 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:03.853705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:03.854127 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:03.854292 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:03.854300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.855263 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.855267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:03.856156 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:03.856166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.856856 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.856860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:03.856972 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:03.857573 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:03.857590 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:03.860117 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":345,"threads_started":1}
W20260925 05:08:03.860540 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:03.852622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:56107 (local address 127.8.215.62:40143)
0925 05:08:03.852837 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:03.852845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:03.853245 (+ 400us) server_negotiation.cc:407] Connection header received
0925 05:08:03.853446 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:03.853451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:03.853524 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:03.853638 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:03.854413 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.855141 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.856274 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.856713 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.856828 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:03.860305 (+ 3477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:56107: BlockingRecv error: recv got EOF from 127.8.215.3:56107 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 05:08:03 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:03 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:03.937702 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.929024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:58639)
0925 05:08:03.929588 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.929607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:03.929772 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:03.930094 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:03.930099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:03.930498 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:03.930655 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:03.930663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.931668 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.931674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:03.932587 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:03.932597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.933425 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.933429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:03.933542 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:03.934167 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:03.934185 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:03.937369 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":249,"threads_started":1}
W20260925 05:08:03.937770 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:03.929079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:58639 (local address 127.8.215.62:40143)
0925 05:08:03.929257 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:03.929261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:03.929693 (+ 432us) server_negotiation.cc:407] Connection header received
0925 05:08:03.929891 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:03.929897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:03.929968 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:03.930067 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:03.930800 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.931531 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.932803 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:03.933290 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:03.933405 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:03.937551 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:58639: BlockingRecv error: recv got EOF from 127.8.215.1:58639 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 05:08:04 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:04 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:04.400254 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:04.392343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:57669)
0925 05:08:04.392899 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:04.392917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:04.393028 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:04.393301 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:04.393304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:04.393685 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:04.393844 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:04.393852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.394847 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.394851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:04.395752 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:04.395762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.396501 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.396506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:04.396616 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:04.397337 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:04.397365 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:04.400007 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":218,"threads_started":1}
W20260925 05:08:04.400403 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:04.392373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:57669 (local address 127.8.215.62:40143)
0925 05:08:04.392583 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:04.392602 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:04.392989 (+ 387us) server_negotiation.cc:407] Connection header received
0925 05:08:04.393120 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:04.393124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:04.393183 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:04.393260 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:04.393960 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.394731 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.395921 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.396378 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.396466 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:04.400180 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:57669: BlockingRecv error: recv got EOF from 127.8.215.2:57669 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 05:08:04 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:04 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:04.869680 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:04.861331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:59861)
0925 05:08:04.862007 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:04.862023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:04.862134 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:04.862418 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:04.862421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:04.862820 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:04.862978 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:04.862986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.863948 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.863952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:04.864909 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:04.864921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.865589 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.865618 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:04.865725 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:04.866320 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:04.866337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:04.869428 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":348,"threads_started":1}
W20260925 05:08:04.869812 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:04.861437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:59861 (local address 127.8.215.62:40143)
0925 05:08:04.861643 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:04.861647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:04.862099 (+ 452us) server_negotiation.cc:407] Connection header received
0925 05:08:04.862218 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:04.862221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:04.862291 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:04.862374 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:04.863102 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.863828 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.865032 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.865460 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.865594 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:04.869611 (+ 4017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:59861: BlockingRecv error: recv got EOF from 127.8.215.3:59861 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 05:08:04 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:04 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:04.948653 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:04.938917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:45527)
0925 05:08:04.940405 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:04.940435 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:04.940592 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:04.941258 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:04.941263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:04.941710 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:04.941855 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:04.941863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.942929 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.942933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:04.943833 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:04.943843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.944588 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.944592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:04.944718 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:04.945354 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:04.945370 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:04.948391 (+ 3021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1309,"thread_start_us":1163,"threads_started":1}
W20260925 05:08:04.948943 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:04.939062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:45527 (local address 127.8.215.62:40143)
0925 05:08:04.939359 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:04.939368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:04.940540 (+ 1172us) server_negotiation.cc:407] Connection header received
0925 05:08:04.940752 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:04.940758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:04.940833 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:04.941042 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:04.942002 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.942796 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.943963 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:04.944435 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:04.944533 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:04.948645 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:45527: BlockingRecv error: recv got EOF from 127.8.215.1:45527 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 05:08:05 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:05 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:05.409499 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:05.401192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:50919)
0925 05:08:05.401702 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:05.401722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:05.401832 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:05.402155 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:05.402158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:05.402533 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:05.402686 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:05.402694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.403922 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.403925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:05.404841 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:05.404851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.405570 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.405574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:05.405691 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:05.406333 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:05.406350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:05.409145 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
W20260925 05:08:05.409608 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:05.401295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50919 (local address 127.8.215.62:40143)
0925 05:08:05.401453 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:05.401457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:05.401819 (+ 362us) server_negotiation.cc:407] Connection header received
0925 05:08:05.401930 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:05.401934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:05.402019 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:05.402117 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:05.402823 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.403787 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.404967 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.405412 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.405530 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:05.409354 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:50919: BlockingRecv error: recv got EOF from 127.8.215.2:50919 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 05:08:05 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:05 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:05.879076 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:05.870861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:39425)
0925 05:08:05.871397 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:05.871413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:05.871584 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:05.871927 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:05.871932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:05.872351 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:05.872494 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:05.872502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.873570 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.873576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:05.874625 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:05.874637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.875466 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.875473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:05.875642 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:05.876280 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:05.876298 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:05.878866 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
W20260925 05:08:05.879285 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:05.870924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:39425 (local address 127.8.215.62:40143)
0925 05:08:05.871174 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:05.871182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:05.871516 (+ 334us) server_negotiation.cc:407] Connection header received
0925 05:08:05.871725 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:05.871730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:05.871789 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:05.871896 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:05.872642 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.873425 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.874767 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.875287 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.875434 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:05.879036 (+ 3602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:39425: BlockingRecv error: recv got EOF from 127.8.215.3:39425 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 05:08:05 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:05 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:05.957350 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:05.949503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:59017)
0925 05:08:05.950036 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:05.950056 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:05.950166 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:05.950468 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:05.950472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:05.950862 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:05.951020 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:05.951027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.951972 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.951975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:05.953105 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:05.953117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.953761 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.953765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:05.953879 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:05.954458 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:05.954474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:05.957095 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
W20260925 05:08:05.957599 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:05.949576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:59017 (local address 127.8.215.62:40143)
0925 05:08:05.949739 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:05.949743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:05.950127 (+ 384us) server_negotiation.cc:407] Connection header received
0925 05:08:05.950288 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:05.950291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:05.950350 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:05.950429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:05.951135 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.951861 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.953229 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:05.953636 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:05.953733 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:05.957361 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:59017: BlockingRecv error: recv got EOF from 127.8.215.1:59017 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 05:08:06 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:06 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:06.419278 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:06.410722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:58577)
0925 05:08:06.411307 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:06.411335 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:06.411448 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:06.411731 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:06.411735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:06.412090 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:06.412276 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:06.412285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.413663 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.413669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:06.414748 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:06.414759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.415408 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.415412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:06.415531 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:06.416161 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:06.416179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:06.419017 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":325,"threads_started":1}
W20260925 05:08:06.419493 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:06.410806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:58577 (local address 127.8.215.62:40143)
0925 05:08:06.410963 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:06.410967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:06.411407 (+ 440us) server_negotiation.cc:407] Connection header received
0925 05:08:06.411541 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:06.411544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:06.411603 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:06.411687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:06.412455 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.413501 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.414876 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.415277 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.415372 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:06.419249 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:58577: BlockingRecv error: recv got EOF from 127.8.215.2:58577 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 05:08:06 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:06 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:06.892757 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:06.880040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:54919)
0925 05:08:06.880558 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:06.880575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:06.880682 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:06.881129 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:06.881133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:06.881488 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:06.881644 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:06.881653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.882579 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.882583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:06.883546 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:06.883560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.884275 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.884281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:06.884456 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:06.889201 (+ 4745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:06.889225 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:06.892386 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
W20260925 05:08:06.892832 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:06.880084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:54919 (local address 127.8.215.62:40143)
0925 05:08:06.880259 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:06.880263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:06.880657 (+ 394us) server_negotiation.cc:407] Connection header received
0925 05:08:06.880914 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:06.880922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:06.881006 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:06.881082 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:06.881768 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.882472 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.883696 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.884125 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.884270 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:06.892567 (+ 8297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:54919: BlockingRecv error: recv got EOF from 127.8.215.3:54919 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 05:08:06 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:06 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:06.966339 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:06.958167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:49603)
0925 05:08:06.958618 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:06.958642 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:06.958748 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:06.959017 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:06.959020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:06.959378 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:06.959515 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:06.959523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.960449 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.960453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:06.961683 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:06.961695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.962634 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.962639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:06.962754 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:06.963281 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:06.963297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:06.966104 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":264,"threads_started":1}
W20260925 05:08:06.966457 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:06.958328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:49603 (local address 127.8.215.62:40143)
0925 05:08:06.958553 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:06.958559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:06.958714 (+ 155us) server_negotiation.cc:407] Connection header received
0925 05:08:06.958840 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:06.958843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:06.958901 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:06.958978 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:06.959630 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.960334 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.961809 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:06.962485 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:06.962591 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:06.966274 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:49603: BlockingRecv error: recv got EOF from 127.8.215.1:49603 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 05:08:07 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:07 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:07.428090 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:07.420247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:56221)
0925 05:08:07.420816 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:07.420838 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:07.420943 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:07.421215 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:07.421218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:07.421561 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:07.421728 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:07.421735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.422683 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.422687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:07.423560 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:07.423570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.424206 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.424210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:07.424323 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:07.424936 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:07.424951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:07.427769 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
W20260925 05:08:07.428236 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:07.420285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:56221 (local address 127.8.215.62:40143)
0925 05:08:07.420453 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:07.420457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:07.420908 (+ 451us) server_negotiation.cc:407] Connection header received
0925 05:08:07.421035 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:07.421039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:07.421094 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:07.421171 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:07.421847 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.422548 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.423684 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.424087 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.424174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:07.427992 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:56221: BlockingRecv error: recv got EOF from 127.8.215.2:56221 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 05:08:07 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:07 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:07.902477 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:07.893738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:39245)
0925 05:08:07.894277 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:07.894292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:07.894396 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:07.894687 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:07.894690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:07.895059 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:07.895202 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:07.895210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.896215 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.896219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:07.897318 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:07.897330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.898023 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.898027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:07.898150 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:07.898827 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:07.898851 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:07.902263 (+ 3412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":299,"threads_started":1}
W20260925 05:08:07.902710 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:07.893891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:39245 (local address 127.8.215.62:40143)
0925 05:08:07.894097 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:07.894102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:07.894360 (+ 258us) server_negotiation.cc:407] Connection header received
0925 05:08:07.894488 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:07.894491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:07.894550 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:07.894631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:07.895328 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.896090 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.897457 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.897897 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.897994 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:07.902458 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:39245: BlockingRecv error: recv got EOF from 127.8.215.3:39245 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 05:08:07 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:07 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:07.976799 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:07.967347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:54207)
0925 05:08:07.967889 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:07.967910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:07.968072 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:07.968447 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:07.968452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:07.969001 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:07.969211 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:07.969245 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.970707 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.970715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:07.971729 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:07.971741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.972646 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.972653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:07.972879 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:07.973457 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:07.973475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:07.976381 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":292,"threads_started":1}
W20260925 05:08:07.976881 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:07.967495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:54207 (local address 127.8.215.62:40143)
0925 05:08:07.967764 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:07.967770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:07.967997 (+ 227us) server_negotiation.cc:407] Connection header received
0925 05:08:07.968222 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:07.968229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:07.968309 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:07.968416 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:07.969393 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.970562 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.971875 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:07.972374 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:07.972636 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:07.976598 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:54207: BlockingRecv error: recv got EOF from 127.8.215.1:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 25 05:08:08 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:08 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:08.437580 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:08.429055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:39031)
0925 05:08:08.429585 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:08.429605 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:08.429708 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:08.430001 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:08.430004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:08.430397 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:08.430561 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:08.430569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.431715 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.431719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:08.432800 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:08.432812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.433690 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.433694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:08.433885 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:08.434555 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:08.434571 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:08.437354 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
W20260925 05:08:08.437796 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:08.429171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:39031 (local address 127.8.215.62:40143)
0925 05:08:08.429329 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:08.429333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:08.429676 (+ 343us) server_negotiation.cc:407] Connection header received
0925 05:08:08.429800 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:08.429803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:08.429859 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:08.429955 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:08.430678 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.431595 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.432947 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.433528 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.433668 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:08.437545 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:39031: BlockingRecv error: recv got EOF from 127.8.215.2:39031 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 05:08:08 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:08 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:08.911511 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:08.903352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:42361)
0925 05:08:08.903846 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:08.903861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:08.904018 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:08.904345 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:08.904349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:08.904792 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:08.904959 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:08.904967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.906074 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.906083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:08.907071 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:08.907082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.907802 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.907808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:08.907983 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:08.908589 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:08.908605 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:08.911248 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
W20260925 05:08:08.911684 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:08.903426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:42361 (local address 127.8.215.62:40143)
0925 05:08:08.903680 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:08.903686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:08.903950 (+ 264us) server_negotiation.cc:407] Connection header received
0925 05:08:08.904143 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:08.904148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:08.904215 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:08.904315 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:08.905103 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.905952 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.907216 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.907641 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.907789 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:08.911450 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:42361: BlockingRecv error: recv got EOF from 127.8.215.3:42361 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 05:08:08 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:08 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:08.985937 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:08.977819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:58881)
0925 05:08:08.978364 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:08.978391 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:08.978536 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:08.978887 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:08.978893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:08.979325 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:08.979467 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:08.979475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.980495 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.980502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:08.981501 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:08.981512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.982265 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.982271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:08.982447 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:08.983005 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:08.983021 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:08.985611 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":301,"threads_started":1}
W20260925 05:08:08.986030 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:08.977984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:58881 (local address 127.8.215.62:40143)
0925 05:08:08.978173 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:08.978180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:08.978477 (+ 297us) server_negotiation.cc:407] Connection header received
0925 05:08:08.978658 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:08.978664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:08.978733 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:08.978832 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:08.979607 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.980357 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.981657 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:08.982122 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:08.982261 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:08.985815 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:58881: BlockingRecv error: recv got EOF from 127.8.215.1:58881 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 05:08:09 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:09 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:09.447310 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:09.438581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:43683)
0925 05:08:09.439208 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:09.439255 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:09.439390 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:09.439687 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:09.439690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:09.440048 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:09.440220 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:09.440228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.441524 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.441530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:09.442762 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:09.442773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.443514 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.443521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:09.443679 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:09.444333 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:09.444351 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:09.447086 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":348,"threads_started":1}
W20260925 05:08:09.447506 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:09.438630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:43683 (local address 127.8.215.62:40143)
0925 05:08:09.438862 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:09.438872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:09.439338 (+ 466us) server_negotiation.cc:407] Connection header received
0925 05:08:09.439489 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:09.439492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:09.439553 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:09.439652 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:09.440334 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.441374 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.442877 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.443312 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.443464 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:09.447267 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:43683: BlockingRecv error: recv got EOF from 127.8.215.2:43683 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 05:08:09 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:09 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:09.920711 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:09.912534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:57561)
0925 05:08:09.913131 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:09.913148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:09.913259 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:09.913545 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:09.913548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:09.913911 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:09.914061 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:09.914069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.915056 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.915059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:09.915956 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:09.915967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.916591 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.916594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:09.916714 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:09.917449 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:09.917476 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:09.920487 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":351,"threads_started":1}
W20260925 05:08:09.921262 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:09.912584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:57561 (local address 127.8.215.62:40143)
0925 05:08:09.912788 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:09.912792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:09.913221 (+ 429us) server_negotiation.cc:407] Connection header received
0925 05:08:09.913355 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:09.913358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:09.913416 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:09.913495 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:09.914177 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.914930 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.916070 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.916460 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.916582 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:09.920843 (+ 4261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:57561: BlockingRecv error: recv got EOF from 127.8.215.3:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 05:08:09 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:09 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:09.995563 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:09.987132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:48539)
0925 05:08:09.987696 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:09.987715 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:09.987840 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:09.988224 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:09.988229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:09.988636 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:09.988830 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:09.988839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.989947 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.989954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:09.991046 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:09.991057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.991805 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.991812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:09.991966 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:09.992588 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:09.992604 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:09.995343 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":318,"threads_started":1}
W20260925 05:08:09.995873 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:09.987232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:48539 (local address 127.8.215.62:40143)
0925 05:08:09.987414 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:09.987418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:09.987804 (+ 386us) server_negotiation.cc:407] Connection header received
0925 05:08:09.987985 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:09.987989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:09.988075 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:09.988192 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:09.988976 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.989820 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.991184 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:09.991613 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:09.991748 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:09.995616 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:48539: BlockingRecv error: recv got EOF from 127.8.215.1:48539 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 05:08:10 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:10 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:10.455859 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:10.448239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:41867)
0925 05:08:10.448802 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:10.448822 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:10.448930 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:10.449190 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:10.449193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:10.449543 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:10.449689 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:10.449696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.450687 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.450690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:10.451590 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:10.451600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.452253 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.452257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:10.452371 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:10.452975 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:10.452992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:10.455623 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":266,"threads_started":1}
W20260925 05:08:10.456007 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:10.448365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:41867 (local address 127.8.215.62:40143)
0925 05:08:10.448592 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:10.448598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:10.448879 (+ 281us) server_negotiation.cc:407] Connection header received
0925 05:08:10.449011 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:10.449014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:10.449072 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:10.449165 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:10.449827 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.450576 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.451708 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.452095 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.452191 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:10.455809 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:41867: BlockingRecv error: recv got EOF from 127.8.215.2:41867 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 05:08:10 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:10 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:10.929756 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:10.921639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:54553)
0925 05:08:10.922190 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:10.922205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:10.922333 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:10.922625 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:10.922628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:10.922982 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:10.923132 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:10.923140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.924193 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.924196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:10.925209 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:10.925221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.925926 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.925930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:10.926046 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:10.926634 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:10.926650 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:10.929525 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
W20260925 05:08:10.929994 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:10.921760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:54553 (local address 127.8.215.62:40143)
0925 05:08:10.921938 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:10.921942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:10.922323 (+ 381us) server_negotiation.cc:407] Connection header received
0925 05:08:10.922432 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:10.922436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:10.922498 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:10.922595 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:10.923287 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.924041 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.925359 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.925774 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.925873 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:10.929755 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:54553: BlockingRecv error: recv got EOF from 127.8.215.3:54553 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 05:08:10 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:10 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:11.005069 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:10.996637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:56043)
0925 05:08:10.997184 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:10.997214 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:10.997329 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:10.997595 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:10.997599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:10.997960 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:10.998131 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:10.998139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.999195 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:10.999199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:11.000081 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:11.000091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.000894 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.000898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:11.001015 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:11.001582 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:11.001599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:11.004691 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":288,"threads_started":1}
W20260925 05:08:11.005151 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:10.996703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:56043 (local address 127.8.215.62:40143)
0925 05:08:10.997044 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:10.997050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:10.997297 (+ 247us) server_negotiation.cc:407] Connection header received
0925 05:08:10.997412 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:10.997415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:10.997475 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:10.997553 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:10.998286 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:10.999055 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.000218 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.000692 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.000879 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:11.004911 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:56043: BlockingRecv error: recv got EOF from 127.8.215.1:56043 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
Sep 25 05:08:11 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:11 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:11.464485 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:11.456846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:42611)
0925 05:08:11.457255 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:11.457277 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:11.457385 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:11.457660 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:11.457663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:11.458012 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:11.458186 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:11.458194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.459166 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.459170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:11.460062 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:11.460072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.460877 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.460881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:11.460996 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:11.461586 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:11.461602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:11.464239 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
W20260925 05:08:11.464617 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:11.456992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:42611 (local address 127.8.215.62:40143)
0925 05:08:11.457222 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:11.457230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:11.457352 (+ 122us) server_negotiation.cc:407] Connection header received
0925 05:08:11.457482 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:11.457485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:11.457544 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:11.457621 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:11.458330 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.459048 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.460226 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.460753 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.460855 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:11.464424 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:42611: BlockingRecv error: recv got EOF from 127.8.215.2:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 05:08:11 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:11 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.3@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:11.946061 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:11.930634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.3:39599)
0925 05:08:11.931281 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:11.931300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:11.931501 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:11.931906 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:11.931911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:11.932348 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:11.932557 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:11.932568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.933773 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.933781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:11.934995 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:11.935009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.935846 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.935853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:11.936076 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:11.942443 (+ 6367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:11.942462 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:11.945608 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":329,"threads_started":1}
W20260925 05:08:11.946173 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:11.930824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:39599 (local address 127.8.215.62:40143)
0925 05:08:11.931067 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:11.931077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:11.931403 (+ 326us) server_negotiation.cc:407] Connection header received
0925 05:08:11.931673 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:11.931679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:11.931747 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:11.931873 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:11.932747 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.933589 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.935172 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:11.935675 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:11.935838 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:11.945873 (+ 10035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.3:39599: BlockingRecv error: recv got EOF from 127.8.215.3:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 05:08:12 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:12 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.1@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:12.015491 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.006288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.1:50239)
0925 05:08:12.006793 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:12.006814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:12.006922 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:12.007222 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:12.007225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:12.007594 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:12.007792 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:12.007801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.009090 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.009097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:12.010163 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:12.010177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.011034 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.011042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:12.011263 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.011881 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:12.011901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:12.015029 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":292,"threads_started":1}
W20260925 05:08:12.015568 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:12.006335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:50239 (local address 127.8.215.62:40143)
0925 05:08:12.006666 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:12.006674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:12.006901 (+ 227us) server_negotiation.cc:407] Connection header received
0925 05:08:12.007027 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:12.007033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:12.007102 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:12.007191 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:12.007955 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.008902 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.010330 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.010841 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.011009 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.015258 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.1:50239: BlockingRecv error: recv got EOF from 127.8.215.1:50239 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 05:08:12 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:08:12 dist-test-slave-wpdm krb5kdc[17297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.215.2@KRBTEST.COM for oryx/127.8.215.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:08:12.474436 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.465647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:40143 (local address 127.8.215.2:35727)
0925 05:08:12.466066 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:12.466085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:12.466192 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:12.466478 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:12.466481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:12.466854 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:12.467030 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:12.467037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.467965 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.467968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:12.468887 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:12.468897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.469731 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.469735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:12.469847 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.470488 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:12.470509 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:12.474012 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.215.62:40143: Server oryx/127.8.215.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":239,"threads_started":1}
W20260925 05:08:12.474484 18035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:12.465692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:35727 (local address 127.8.215.62:40143)
0925 05:08:12.465911 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:12.465918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:12.466184 (+ 266us) server_negotiation.cc:407] Connection header received
0925 05:08:12.466304 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:12.466308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:12.466367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:12.466439 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:12.467152 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.467848 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.469062 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.469597 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.469723 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.474220 (+ 4497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.215.2:35727: BlockingRecv error: recv got EOF from 127.8.215.2:35727 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:08:12.768716 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17886
I20260925 05:08:12.780344 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.769770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.8.215.1:45331)
0925 05:08:12.770350 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:12.770358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:12.770385 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:08:12.770448 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:12.770454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:12.770674 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:12.770829 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:12.772150 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.773239 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.774396 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.774916 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.775009 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.777421 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:12.777430 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:12.777443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:12.777472 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:12.779861 (+ 2389us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.215.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 05:08:12.780100 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60534: Request ticket server kudu/127.8.215.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
W20260925 05:08:12.780449 18346 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60534: Request ticket server kudu/127.8.215.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 05:08:12.780506 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17886: Not authorized: Client connection negotiation failed: client connection to 127.8.215.1:45331: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.215.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 05:08:12.780786 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17886
I20260925 05:08:12.798349 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18037
I20260925 05:08:12.810194 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.799399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.8.215.2:39473)
0925 05:08:12.799970 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:12.799976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:12.800005 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:08:12.800060 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:12.800067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:12.800290 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:12.800431 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:12.801689 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.802725 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.804025 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.804578 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.804674 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.807097 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:12.807105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:12.807119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:12.807152 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:12.809653 (+ 2501us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.215.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 05:08:12.809891 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58240: Request ticket server kudu/127.8.215.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
W20260925 05:08:12.810320 18347 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58240: Request ticket server kudu/127.8.215.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 05:08:12.810290 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18037: Not authorized: Client connection negotiation failed: client connection to 127.8.215.2:39473: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.215.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 05:08:12.810460 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18037
I20260925 05:08:12.826929 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18168
I20260925 05:08:12.838948 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.827890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.8.215.3:39449)
0925 05:08:12.828432 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:12.828438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:12.828464 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:08:12.828521 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:12.828527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:12.828801 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:12.828958 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:12.830199 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.831815 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.833022 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:12.833556 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:12.833653 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:12.835962 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:12.835970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:12.835983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:12.836029 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:12.838455 (+ 2426us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.215.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 05:08:12.838683 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48010: Request ticket server kudu/127.8.215.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
W20260925 05:08:12.839048 18348 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48010: Request ticket server kudu/127.8.215.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 05:08:12.839083 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18168: Not authorized: Client connection negotiation failed: client connection to 127.8.215.3:39449: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.215.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 05:08:12.839274 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18168
I20260925 05:08:12.856076 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 17817
I20260925 05:08:12.857420 17879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:12.974869 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 17817
2026-09-25T05:08:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21202 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T05:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:13Z Disabled control of system clock
I20260925 05:08:13.053557 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43509
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:08:13.376564 18353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:13.376965 18353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:13.377059 18353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:13.384495 18353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:13.384608 18353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:13.384658 18353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:13.384713 18353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:13.384784 18353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:13.394177 18353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:13.395444 18353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:13.397217 18353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:13.409602 18358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:13.409576 18359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:13.409696 18361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:13.410198 18353 server_base.cc:1034] running on GCE node
I20260925 05:08:13.411068 18353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:13.412539 18353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:13.413772 18353 hybrid_clock.cc:648] HybridClock initialized: now 1790312893413730 us; error 56 us; skew 500 ppm
I20260925 05:08:13.414130 18353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:13.416410 18353 webserver.cc:460] Webserver started at http://127.8.215.62:39373/ using document root <none> and password file <none>
I20260925 05:08:13.417168 18353 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:13.417310 18353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:13.417631 18353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:13.420466 18353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "f6d7f212f53843e19191f622bc75ea76"
format_stamp: "Formatted at 2026-09-25 05:08:13 on dist-test-slave-wpdm"
server_key: "58be76c284ccb7273c328724e3912f82"
server_key_iv: "81dd24d199e11f579e6104e4e1e39c80"
server_key_version: "encryptionkey@0"
I20260925 05:08:13.421324 18353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "f6d7f212f53843e19191f622bc75ea76"
format_stamp: "Formatted at 2026-09-25 05:08:13 on dist-test-slave-wpdm"
server_key: "58be76c284ccb7273c328724e3912f82"
server_key_iv: "81dd24d199e11f579e6104e4e1e39c80"
server_key_version: "encryptionkey@0"
I20260925 05:08:13.426512 18353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:08:13.429989 18367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:13.431526 18353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:08:13.431782 18353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "f6d7f212f53843e19191f622bc75ea76"
format_stamp: "Formatted at 2026-09-25 05:08:13 on dist-test-slave-wpdm"
server_key: "58be76c284ccb7273c328724e3912f82"
server_key_iv: "81dd24d199e11f579e6104e4e1e39c80"
server_key_version: "encryptionkey@0"
I20260925 05:08:13.432054 18353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:13.446218 18353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:13.447072 18353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:13.447358 18353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:13.465793 18353 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43509
I20260925 05:08:13.465811 18418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43509 every 8 connection(s)
I20260925 05:08:13.467191 18353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:13.470933 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:13.474131 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18353
I20260925 05:08:13.474347 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:08:13.474970 9052 external_mini_cluster.cc:1428] Setting key 72945ce8aee69d0d1618ad0ec9bb05a8
I20260925 05:08:13.484433 18419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76: Bootstrap starting.
I20260925 05:08:13.488234 18419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:13.489425 18419 log.cc:824] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:13.489897 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:13.477027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.8.215.62:43509)
0925 05:08:13.478083 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:13.478098 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:13.478162 (+ 64us) server_negotiation.cc:407] Connection header received
0925 05:08:13.479947 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:13.479982 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:13.480629 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:13.481114 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:13.481666 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:13.483608 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:13.485024 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:13.486023 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:13.486262 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:13.487340 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:13.487352 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:13.487371 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:13.488028 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:13.488161 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:13.488187 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:13.488312 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:13.488585 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 05:08:13.488928 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":317,"threads_started":1}
I20260925 05:08:13.492175 18419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76: No bootstrap required, opened a new log
I20260925 05:08:13.499715 18419 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d7f212f53843e19191f622bc75ea76" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43509 } }
I20260925 05:08:13.500080 18419 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:13.500185 18419 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d7f212f53843e19191f622bc75ea76, State: Initialized, Role: FOLLOWER
I20260925 05:08:13.500777 18419 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d7f212f53843e19191f622bc75ea76" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43509 } }
I20260925 05:08:13.501017 18419 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:13.501152 18419 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:13.501335 18419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:13.503480 18419 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d7f212f53843e19191f622bc75ea76" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43509 } }
I20260925 05:08:13.503979 18419 leader_election.cc:302] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d7f212f53843e19191f622bc75ea76; no voters:
I20260925 05:08:13.504658 18419 leader_election.cc:288] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:08:13.504815 18424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:13.505650 18424 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [term 1 LEADER]: Becoming Leader. State: Replica: f6d7f212f53843e19191f622bc75ea76, State: Running, Role: LEADER
I20260925 05:08:13.506165 18424 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d7f212f53843e19191f622bc75ea76" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43509 } }
I20260925 05:08:13.506733 18419 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:13.509369 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d7f212f53843e19191f622bc75ea76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d7f212f53843e19191f622bc75ea76" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43509 } } }
I20260925 05:08:13.509817 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:13.513130 18426 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d7f212f53843e19191f622bc75ea76. Latest consensus state: current_term: 1 leader_uuid: "f6d7f212f53843e19191f622bc75ea76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d7f212f53843e19191f622bc75ea76" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43509 } } }
I20260925 05:08:13.513501 18426 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:13.513705 18431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:13.518667 18431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:13.527426 18431 catalog_manager.cc:1347] Generated new cluster ID: ca35f1294a2f4c83b0c1a1d6148b159c
I20260925 05:08:13.527536 18431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:08:13.545800 18431 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:08:13.547573 18431 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:13.563369 18431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6d7f212f53843e19191f622bc75ea76: Generated new TSK 0
I20260925 05:08:13.564443 18431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:08:13.577914 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43509
--builtin_ntp_servers=127.8.215.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:08:13.912186 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:13.912497 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:13.912660 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:13.919797 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:13.919883 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:13.920042 18443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:08:13.929045 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:13.930277 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:13.931995 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:13.945248 18448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:13.945204 18449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:13.945298 18451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:13.950174 18443 server_base.cc:1034] running on GCE node
I20260925 05:08:13.950871 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:13.951731 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:13.952927 18443 hybrid_clock.cc:648] HybridClock initialized: now 1790312893952884 us; error 31 us; skew 500 ppm
I20260925 05:08:13.953310 18443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:13.955459 18443 webserver.cc:460] Webserver started at http://127.8.215.1:36001/ using document root <none> and password file <none>
I20260925 05:08:13.956169 18443 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:13.956292 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:13.956614 18443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:13.959338 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "d5671d75620e4b3b80a57ca9eac32743"
format_stamp: "Formatted at 2026-09-25 05:08:13 on dist-test-slave-wpdm"
server_key: "10201b8d6e8e1f9b785ae5d7e263bc10"
server_key_iv: "824047c804746ea9f2f6bfbe0a6c635a"
server_key_version: "encryptionkey@0"
I20260925 05:08:13.960093 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "d5671d75620e4b3b80a57ca9eac32743"
format_stamp: "Formatted at 2026-09-25 05:08:13 on dist-test-slave-wpdm"
server_key: "10201b8d6e8e1f9b785ae5d7e263bc10"
server_key_iv: "824047c804746ea9f2f6bfbe0a6c635a"
server_key_version: "encryptionkey@0"
I20260925 05:08:13.964918 18443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 05:08:13.968137 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:13.969420 18443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:13.969581 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "d5671d75620e4b3b80a57ca9eac32743"
format_stamp: "Formatted at 2026-09-25 05:08:13 on dist-test-slave-wpdm"
server_key: "10201b8d6e8e1f9b785ae5d7e263bc10"
server_key_iv: "824047c804746ea9f2f6bfbe0a6c635a"
server_key_version: "encryptionkey@0"
I20260925 05:08:13.969803 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:13.995729 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:13.996613 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:13.996923 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:13.997923 18443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:13.999338 18443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:13.999425 18443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:13.999531 18443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:13.999634 18443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.033592 18443 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:34703
I20260925 05:08:14.033689 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:34703 every 8 connection(s)
I20260925 05:08:14.034927 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:14.041603 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18443
I20260925 05:08:14.041788 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:08:14.042603 9052 external_mini_cluster.cc:1428] Setting key 3a0a31a744a435b15270cffdc849963a
I20260925 05:08:14.045372 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43509
--builtin_ntp_servers=127.8.215.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 05:08:14.049286 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.037453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:42053 (local address 127.8.215.62:43509)
0925 05:08:14.037922 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:14.037928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:14.039117 (+ 1189us) server_negotiation.cc:407] Connection header received
0925 05:08:14.040848 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:14.040852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:14.040913 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:14.041007 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:14.042083 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.043195 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.045094 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.045675 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.045812 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.047566 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:14.047574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:14.047608 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:14.047634 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:14.047710 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:14.047728 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:14.047880 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:14.048813 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 05:08:14.048925 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260925 05:08:14.050357 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.038101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43509 (local address 127.8.215.1:42053)
0925 05:08:14.038972 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:14.039016 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:14.040546 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:14.041212 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:14.041220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:14.041307 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:14.041911 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:14.041936 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.043361 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.043366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:14.044956 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:14.044967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.045907 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.045916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:14.046286 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.047193 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:08:14.047199 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:14.047252 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:08:14.047358 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:14.047853 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:14.047866 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:14.047874 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:08:14.048092 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 05:08:14.048590 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":269,"threads_started":1}
I20260925 05:08:14.051853 18570 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43509
I20260925 05:08:14.052158 18570 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:14.053014 18570 heartbeater.cc:511] Master 127.8.215.62:43509 requested a full tablet report, sending...
I20260925 05:08:14.055389 18384 ts_manager.cc:194] Registered new tserver with Master: d5671d75620e4b3b80a57ca9eac32743 (127.8.215.1:34703)
I20260925 05:08:14.057591 18384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.1:42053
W20260925 05:08:14.371156 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:14.371436 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:14.371519 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:14.379055 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:14.379151 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:14.379254 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:08:14.388147 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:14.389469 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:14.391146 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:14.405102 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:14.405171 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:14.405201 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:14.406157 18574 server_base.cc:1034] running on GCE node
I20260925 05:08:14.406692 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:14.407626 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:14.408830 18574 hybrid_clock.cc:648] HybridClock initialized: now 1790312894408798 us; error 32 us; skew 500 ppm
I20260925 05:08:14.409175 18574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:14.411389 18574 webserver.cc:460] Webserver started at http://127.8.215.2:41419/ using document root <none> and password file <none>
I20260925 05:08:14.412088 18574 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:14.412202 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:14.412536 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:14.415469 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "4972d6d9a60e40478fe2984682e55c98"
format_stamp: "Formatted at 2026-09-25 05:08:14 on dist-test-slave-wpdm"
server_key: "f14a1786b2bed5ff3efe1f9e5ca8673a"
server_key_iv: "6a706970f80e1b21b7f5b4c8babbb574"
server_key_version: "encryptionkey@0"
I20260925 05:08:14.416178 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "4972d6d9a60e40478fe2984682e55c98"
format_stamp: "Formatted at 2026-09-25 05:08:14 on dist-test-slave-wpdm"
server_key: "f14a1786b2bed5ff3efe1f9e5ca8673a"
server_key_iv: "6a706970f80e1b21b7f5b4c8babbb574"
server_key_version: "encryptionkey@0"
I20260925 05:08:14.421068 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:08:14.424314 18588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.425727 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:14.425902 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "4972d6d9a60e40478fe2984682e55c98"
format_stamp: "Formatted at 2026-09-25 05:08:14 on dist-test-slave-wpdm"
server_key: "f14a1786b2bed5ff3efe1f9e5ca8673a"
server_key_iv: "6a706970f80e1b21b7f5b4c8babbb574"
server_key_version: "encryptionkey@0"
I20260925 05:08:14.426119 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:14.454247 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:14.455129 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:14.455407 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:14.456395 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:14.457818 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:14.457901 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.458065 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:14.458150 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.491806 18574 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:46161
I20260925 05:08:14.491856 18700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:46161 every 8 connection(s)
I20260925 05:08:14.493100 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:08:14.501142 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18574
I20260925 05:08:14.501317 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:08:14.501859 9052 external_mini_cluster.cc:1428] Setting key db603dac9894ffd514d435b476824d10
I20260925 05:08:14.504495 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43509
--builtin_ntp_servers=127.8.215.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 05:08:14.505602 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.495520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:39557 (local address 127.8.215.62:43509)
0925 05:08:14.495731 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:14.495737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:14.497122 (+ 1385us) server_negotiation.cc:407] Connection header received
0925 05:08:14.498863 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:14.498867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:14.498929 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:14.499015 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:14.500000 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.500693 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.502830 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.503231 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.503372 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.504822 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:14.504827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:14.504831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:14.504847 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:14.504874 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:14.504887 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:14.504936 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:14.505203 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 05:08:14.505304 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:08:14.506655 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.496049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43509 (local address 127.8.215.2:39557)
0925 05:08:14.496975 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:14.497009 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:14.498617 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:14.499214 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:14.499221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:14.499309 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:14.499861 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:14.499877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.500881 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.500888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:14.502701 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:14.502713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.503458 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.503465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:14.503690 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.504408 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:08:14.504411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:14.504448 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:08:14.504524 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:14.505003 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:14.505011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:14.505016 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:08:14.505198 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 05:08:14.505574 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":266,"threads_started":1}
I20260925 05:08:14.508066 18701 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43509
I20260925 05:08:14.508375 18701 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:14.509272 18701 heartbeater.cc:511] Master 127.8.215.62:43509 requested a full tablet report, sending...
I20260925 05:08:14.510744 18384 ts_manager.cc:194] Registered new tserver with Master: 4972d6d9a60e40478fe2984682e55c98 (127.8.215.2:46161)
I20260925 05:08:14.511920 18384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.2:39557
W20260925 05:08:14.828996 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:14.829308 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:14.829394 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:14.836794 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:14.836897 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:14.836998 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:08:14.845906 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:14.847204 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:14.849040 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:14.862182 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:14.862185 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:14.862133 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:14.863063 18704 server_base.cc:1034] running on GCE node
I20260925 05:08:14.863633 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:14.864516 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:14.865706 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790312894865634 us; error 60 us; skew 500 ppm
I20260925 05:08:14.865990 18704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:14.873436 18704 webserver.cc:460] Webserver started at http://127.8.215.3:39935/ using document root <none> and password file <none>
I20260925 05:08:14.874255 18704 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:14.874377 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:14.874655 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:14.877787 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "4d447754510a42f680a723b1344afcc4"
format_stamp: "Formatted at 2026-09-25 05:08:14 on dist-test-slave-wpdm"
server_key: "52777911ef232820b0af92831886aed7"
server_key_iv: "23e6b6bbe0b7bc85b47496a91e8d2e02"
server_key_version: "encryptionkey@0"
I20260925 05:08:14.878588 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "4d447754510a42f680a723b1344afcc4"
format_stamp: "Formatted at 2026-09-25 05:08:14 on dist-test-slave-wpdm"
server_key: "52777911ef232820b0af92831886aed7"
server_key_iv: "23e6b6bbe0b7bc85b47496a91e8d2e02"
server_key_version: "encryptionkey@0"
I20260925 05:08:14.883879 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:08:14.887351 18718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.888870 18704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:14.889029 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "4d447754510a42f680a723b1344afcc4"
format_stamp: "Formatted at 2026-09-25 05:08:14 on dist-test-slave-wpdm"
server_key: "52777911ef232820b0af92831886aed7"
server_key_iv: "23e6b6bbe0b7bc85b47496a91e8d2e02"
server_key_version: "encryptionkey@0"
I20260925 05:08:14.889246 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:14.911414 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:14.912266 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:14.912577 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:14.913726 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:14.915128 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:14.915241 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.915346 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:14.915444 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:14.953316 18704 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:37881
I20260925 05:08:14.953413 18830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:37881 every 8 connection(s)
I20260925 05:08:14.954670 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:08:14.959460 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18704
I20260925 05:08:14.959628 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:08:14.960088 9052 external_mini_cluster.cc:1428] Setting key 785d533bc509020a9a85b8a932ac84fd
I20260925 05:08:14.967335 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.957185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:36301 (local address 127.8.215.62:43509)
0925 05:08:14.957407 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:14.957413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:14.958800 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 05:08:14.960719 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:14.960724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:14.960856 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:14.960941 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:14.962174 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.963188 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.964625 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.965066 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.965164 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.966674 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:14.966678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:14.966682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:14.966699 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:14.966728 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:14.966739 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:14.966815 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:14.967008 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 05:08:14.967079 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:08:14.968396 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.957774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43509 (local address 127.8.215.3:36301)
0925 05:08:14.958653 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:14.958696 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:14.960410 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:14.961227 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:14.961238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:14.961359 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:14.962009 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:14.962040 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.963320 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.963324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:14.964513 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:14.964524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.965257 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.965262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:14.965745 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.966387 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:08:14.966390 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:14.966427 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:08:14.966506 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:14.966831 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:14.966838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:14.966844 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:08:14.966986 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 05:08:14.967285 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260925 05:08:14.969785 18831 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43509
I20260925 05:08:14.970041 18831 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:14.970705 18831 heartbeater.cc:511] Master 127.8.215.62:43509 requested a full tablet report, sending...
I20260925 05:08:14.971992 18384 ts_manager.cc:194] Registered new tserver with Master: 4d447754510a42f680a723b1344afcc4 (127.8.215.3:37881)
I20260925 05:08:14.972836 18384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.3:36301
I20260925 05:08:14.975301 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:08:14.981199 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:14.979718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.8.215.62:43509)
0925 05:08:14.979865 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:14.979869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:14.980317 (+ 448us) server_negotiation.cc:407] Connection header received
0925 05:08:14.980436 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:14.980440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:14.980505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:14.980577 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:14.981029 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38620: BlockingRecv error: recv got EOF from 127.0.0.1:38620 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:08:14.981784 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18443
I20260925 05:08:14.989530 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.982598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.8.215.1:34703)
0925 05:08:14.983221 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:14.983228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:14.983257 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:08:14.983314 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:14.983320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:14.983517 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:14.983671 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:14.984080 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.985104 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.986336 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:14.986887 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:14.987037 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:14.988093 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:14.988101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:14.988113 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:14.988670 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:14.988713 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:14.988778 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:14.988900 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:14.989087 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 05:08:14.989214 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":318,"threads_started":1}
I20260925 05:08:14.990651 18565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:15.093091 18570 heartbeater.cc:503] Master 127.8.215.62:43509 was elected leader, sending a full tablet report...
I20260925 05:08:15.094449 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18443
W20260925 05:08:15.111218 18368 connection.cc:535] server connection from 127.8.215.1:42053 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:08:15.111397 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18574
I20260925 05:08:15.119333 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:15.112337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.8.215.2:46161)
0925 05:08:15.112889 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:15.112899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:15.112926 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:08:15.112981 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:15.113009 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:15.113256 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:15.113385 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:15.113804 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:15.114913 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:15.116089 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:15.116639 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:15.116845 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:15.117924 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:15.117931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:15.117944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:15.118526 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:15.118570 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:15.118589 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:15.118693 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:15.118828 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 05:08:15.118963 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260925 05:08:15.120764 18696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:15.201392 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18574
I20260925 05:08:15.218024 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18704
I20260925 05:08:15.225697 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:15.218943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.8.215.3:37881)
0925 05:08:15.219531 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:15.219538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:15.219567 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:08:15.219628 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:15.219635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:15.219857 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:15.219988 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:15.220344 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:15.221446 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:15.222647 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:15.223196 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:15.223293 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:15.224307 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:15.224316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:15.224328 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:15.224986 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:15.225037 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:15.225055 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:15.225142 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:15.225270 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:08:15.225387 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":303,"threads_started":1}
I20260925 05:08:15.226962 18826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:15.309453 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18704
I20260925 05:08:15.325634 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18353
I20260925 05:08:15.326813 18414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:15.427100 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18353
2026-09-25T05:08:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2446 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T05:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:15Z Disabled control of system clock
I20260925 05:08:15.499622 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:42743
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:08:15.818084 18850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:15.818426 18850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:15.818570 18850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:15.825752 18850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:15.825899 18850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:15.825976 18850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:15.826085 18850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:15.826165 18850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:15.835242 18850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:42743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:15.836531 18850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:15.838282 18850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:15.849880 18855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:15.849858 18858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:15.850169 18856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:15.850936 18850 server_base.cc:1034] running on GCE node
I20260925 05:08:15.851687 18850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:15.853089 18850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:15.854326 18850 hybrid_clock.cc:648] HybridClock initialized: now 1790312895854287 us; error 55 us; skew 500 ppm
I20260925 05:08:15.854668 18850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:15.857023 18850 webserver.cc:460] Webserver started at http://127.8.215.62:33299/ using document root <none> and password file <none>
I20260925 05:08:15.857714 18850 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:15.857833 18850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:15.858172 18850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:15.860996 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "bd2ecd22457648b9a4d86a9548a0dc90"
format_stamp: "Formatted at 2026-09-25 05:08:15 on dist-test-slave-wpdm"
server_key: "b26524d1acf562549734722f384dc780"
server_key_iv: "e4e9a3a8f7939f5e5c330bdb22cab64f"
server_key_version: "encryptionkey@0"
I20260925 05:08:15.861801 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "bd2ecd22457648b9a4d86a9548a0dc90"
format_stamp: "Formatted at 2026-09-25 05:08:15 on dist-test-slave-wpdm"
server_key: "b26524d1acf562549734722f384dc780"
server_key_iv: "e4e9a3a8f7939f5e5c330bdb22cab64f"
server_key_version: "encryptionkey@0"
I20260925 05:08:15.867252 18850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:08:15.870738 18864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:15.872076 18850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:08:15.872280 18850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "bd2ecd22457648b9a4d86a9548a0dc90"
format_stamp: "Formatted at 2026-09-25 05:08:15 on dist-test-slave-wpdm"
server_key: "b26524d1acf562549734722f384dc780"
server_key_iv: "e4e9a3a8f7939f5e5c330bdb22cab64f"
server_key_version: "encryptionkey@0"
I20260925 05:08:15.872493 18850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:15.877998 18850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:15.878793 18850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:15.879071 18850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:15.896903 18850 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:42743
I20260925 05:08:15.896921 18915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:42743 every 8 connection(s)
I20260925 05:08:15.898203 18850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:15.899217 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18850
I20260925 05:08:15.899437 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:08:15.900241 9052 external_mini_cluster.cc:1428] Setting key 984f0efb86df487ebd1e58051267edaa
I20260925 05:08:15.902855 18916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:15.908419 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:15.902356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.8.215.62:42743)
0925 05:08:15.903354 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:15.903365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:15.903403 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:08:15.904981 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:15.905001 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:15.905390 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:15.905783 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:15.906836 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:15.906842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:15.906853 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:15.906866 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:15.906938 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:15.906952 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:15.907190 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:15.907329 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 05:08:15.907557 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260925 05:08:15.913834 18916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90: Bootstrap starting.
I20260925 05:08:15.916564 18916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:15.917549 18916 log.cc:824] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:15.920245 18916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90: No bootstrap required, opened a new log
I20260925 05:08:15.927127 18916 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42743 } }
I20260925 05:08:15.927522 18916 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:15.927630 18916 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2ecd22457648b9a4d86a9548a0dc90, State: Initialized, Role: FOLLOWER
I20260925 05:08:15.928179 18916 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42743 } }
I20260925 05:08:15.928409 18916 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:15.928555 18916 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:15.928764 18916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:15.930838 18916 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42743 } }
I20260925 05:08:15.931347 18916 leader_election.cc:302] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2ecd22457648b9a4d86a9548a0dc90; no voters:
I20260925 05:08:15.931922 18916 leader_election.cc:288] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:08:15.932068 18921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:15.932754 18921 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [term 1 LEADER]: Becoming Leader. State: Replica: bd2ecd22457648b9a4d86a9548a0dc90, State: Running, Role: LEADER
I20260925 05:08:15.933380 18921 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42743 } }
I20260925 05:08:15.934059 18916 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:15.935839 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2ecd22457648b9a4d86a9548a0dc90. Latest consensus state: current_term: 1 leader_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42743 } } }
I20260925 05:08:15.936182 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:15.937893 18926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:15.937983 18922 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ecd22457648b9a4d86a9548a0dc90" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42743 } } }
I20260925 05:08:15.938426 18922 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:15.943933 18926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:15.952761 18926 catalog_manager.cc:1347] Generated new cluster ID: 38fc03505c2e4f8e800c3e53781bbe2f
I20260925 05:08:15.952872 18926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:08:15.973660 18926 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:08:15.974078 18926 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:15.984565 18926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2ecd22457648b9a4d86a9548a0dc90: Generated new TSK 0
I20260925 05:08:15.985373 18926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:08:15.991927 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42743
--builtin_ntp_servers=127.8.215.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:08:16.320024 18941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:16.320305 18941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:16.320485 18941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:16.327739 18941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:16.327833 18941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:16.328037 18941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:08:16.337405 18941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:16.338716 18941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:16.340540 18941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:16.352636 18946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:16.353219 18947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:16.354557 18949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:16.354931 18941 server_base.cc:1034] running on GCE node
I20260925 05:08:16.355546 18941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:16.356469 18941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:16.357699 18941 hybrid_clock.cc:648] HybridClock initialized: now 1790312896357603 us; error 85 us; skew 500 ppm
I20260925 05:08:16.358016 18941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:16.360249 18941 webserver.cc:460] Webserver started at http://127.8.215.1:36845/ using document root <none> and password file <none>
I20260925 05:08:16.361018 18941 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:16.361150 18941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:16.361474 18941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:16.365782 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "758bd0f6ad5046f68be82b5015940b03"
format_stamp: "Formatted at 2026-09-25 05:08:16 on dist-test-slave-wpdm"
server_key: "76128c9afa8738cd6a22a89dfa38482f"
server_key_iv: "dd8aa745e80a8f297e34e8a7bcaf1102"
server_key_version: "encryptionkey@0"
I20260925 05:08:16.366556 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "758bd0f6ad5046f68be82b5015940b03"
format_stamp: "Formatted at 2026-09-25 05:08:16 on dist-test-slave-wpdm"
server_key: "76128c9afa8738cd6a22a89dfa38482f"
server_key_iv: "dd8aa745e80a8f297e34e8a7bcaf1102"
server_key_version: "encryptionkey@0"
I20260925 05:08:16.371582 18941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:08:16.375017 18955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:16.376410 18941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:08:16.376574 18941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "758bd0f6ad5046f68be82b5015940b03"
format_stamp: "Formatted at 2026-09-25 05:08:16 on dist-test-slave-wpdm"
server_key: "76128c9afa8738cd6a22a89dfa38482f"
server_key_iv: "dd8aa745e80a8f297e34e8a7bcaf1102"
server_key_version: "encryptionkey@0"
I20260925 05:08:16.376861 18941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:16.382382 18941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:16.383188 18941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:16.383459 18941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:16.384331 18941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:16.385888 18941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:16.386016 18941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:16.386144 18941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:16.386233 18941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:16.418318 18941 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:38433
I20260925 05:08:16.418367 19067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:38433 every 8 connection(s)
I20260925 05:08:16.419577 18941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:16.426836 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 18941
I20260925 05:08:16.427050 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:08:16.427527 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.422041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:54345 (local address 127.8.215.62:42743)
0925 05:08:16.422447 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:16.422451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:16.423637 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 05:08:16.425331 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:16.425337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:16.425408 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:16.425522 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:16.426787 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:16.426792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:16.426796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:16.426814 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:16.426861 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:16.426873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:16.426994 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:16.427178 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 05:08:16.427238 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":212,"threads_started":1}
I20260925 05:08:16.427699 9052 external_mini_cluster.cc:1428] Setting key 5c38a6b0d0ad12e7400882b7d0126205
I20260925 05:08:16.428439 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.422644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42743 (local address 127.8.215.1:54345)
0925 05:08:16.423491 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:16.423530 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:16.425082 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:16.425769 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:16.425777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:16.425869 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:16.426496 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:08:16.426498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:16.426532 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:08:16.426612 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:16.426966 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:16.426977 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:16.426984 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:08:16.427255 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 05:08:16.427494 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":265,"threads_started":1}
I20260925 05:08:16.429891 19068 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42743
I20260925 05:08:16.430184 19068 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:16.430387 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42743
--builtin_ntp_servers=127.8.215.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 05:08:16.430786 19068 heartbeater.cc:511] Master 127.8.215.62:42743 requested a full tablet report, sending...
I20260925 05:08:16.432665 18880 ts_manager.cc:194] Registered new tserver with Master: 758bd0f6ad5046f68be82b5015940b03 (127.8.215.1:38433)
W20260925 05:08:16.748539 19072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:16.748870 19072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:16.749033 19072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:16.756136 19072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:16.756269 19072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:16.756448 19072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:08:16.766922 19072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:16.768363 19072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:16.770402 19072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:16.784752 19077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:16.785130 19080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:16.785176 19078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:16.785847 19072 server_base.cc:1034] running on GCE node
I20260925 05:08:16.786509 19072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:16.787421 19072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:16.788664 19072 hybrid_clock.cc:648] HybridClock initialized: now 1790312896788607 us; error 67 us; skew 500 ppm
I20260925 05:08:16.789023 19072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:16.791265 19072 webserver.cc:460] Webserver started at http://127.8.215.2:37699/ using document root <none> and password file <none>
I20260925 05:08:16.791965 19072 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:16.792101 19072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:16.792435 19072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:16.795167 19072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "76a1781bb95d406cad597786c8e5d23d"
format_stamp: "Formatted at 2026-09-25 05:08:16 on dist-test-slave-wpdm"
server_key: "0a9be00422105c73e969f6dbd461b8fb"
server_key_iv: "d569034dbbbe882de055d935ee2e86ad"
server_key_version: "encryptionkey@0"
I20260925 05:08:16.795935 19072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "76a1781bb95d406cad597786c8e5d23d"
format_stamp: "Formatted at 2026-09-25 05:08:16 on dist-test-slave-wpdm"
server_key: "0a9be00422105c73e969f6dbd461b8fb"
server_key_iv: "d569034dbbbe882de055d935ee2e86ad"
server_key_version: "encryptionkey@0"
I20260925 05:08:16.800908 19072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:08:16.804052 19086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:16.805338 19072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:16.805506 19072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "76a1781bb95d406cad597786c8e5d23d"
format_stamp: "Formatted at 2026-09-25 05:08:16 on dist-test-slave-wpdm"
server_key: "0a9be00422105c73e969f6dbd461b8fb"
server_key_iv: "d569034dbbbe882de055d935ee2e86ad"
server_key_version: "encryptionkey@0"
I20260925 05:08:16.805711 19072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:16.811234 19072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:16.811973 19072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:16.812237 19072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:16.813159 19072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:16.814466 19072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:16.814589 19072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:16.814723 19072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:16.814818 19072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:16.847012 19072 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:40925
I20260925 05:08:16.847064 19198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:40925 every 8 connection(s)
I20260925 05:08:16.848354 19072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:08:16.852882 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 19072
I20260925 05:08:16.853075 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:08:16.853502 9052 external_mini_cluster.cc:1428] Setting key 20b1ca2e083a7659c343dcf1fe4b92d1
I20260925 05:08:16.856302 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42743
--builtin_ntp_servers=127.8.215.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 05:08:16.856454 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.850871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:38305 (local address 127.8.215.62:42743)
0925 05:08:16.851032 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:16.851036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:16.852467 (+ 1431us) server_negotiation.cc:407] Connection header received
0925 05:08:16.854268 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:16.854272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:16.854357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:16.854437 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:16.855739 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:16.855745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:16.855751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:16.855773 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:16.855802 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:16.855812 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:16.855933 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:16.856128 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 05:08:16.856189 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:08:16.857404 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.851414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42743 (local address 127.8.215.2:38305)
0925 05:08:16.852306 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:16.852345 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:16.854005 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:16.854660 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:16.854668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:16.854756 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:16.855459 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:08:16.855461 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:16.855501 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:08:16.855583 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:16.855908 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:16.855918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:16.855925 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:08:16.856204 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 05:08:16.856447 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":270,"threads_started":1}
I20260925 05:08:16.858772 19199 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42743
I20260925 05:08:16.859059 19199 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:16.859608 19199 heartbeater.cc:511] Master 127.8.215.62:42743 requested a full tablet report, sending...
I20260925 05:08:16.860921 18880 ts_manager.cc:194] Registered new tserver with Master: 76a1781bb95d406cad597786c8e5d23d (127.8.215.2:40925)
W20260925 05:08:17.197197 19202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:17.197475 19202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:17.197690 19202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:17.204975 19202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:17.205086 19202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:17.205300 19202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:08:17.214532 19202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:17.215806 19202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:17.217715 19202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:17.231552 19207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:17.231498 19210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:17.231528 19208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:17.232586 19202 server_base.cc:1034] running on GCE node
I20260925 05:08:17.233232 19202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:17.234222 19202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:17.235468 19202 hybrid_clock.cc:648] HybridClock initialized: now 1790312897235397 us; error 63 us; skew 500 ppm
I20260925 05:08:17.235793 19202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:08:17.242969 19202 webserver.cc:460] Webserver started at http://127.8.215.3:45301/ using document root <none> and password file <none>
I20260925 05:08:17.243667 19202 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:17.243793 19202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:17.244123 19202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:17.246982 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "15483d368c534a28b0f2904cae335788"
format_stamp: "Formatted at 2026-09-25 05:08:17 on dist-test-slave-wpdm"
server_key: "28e92e967a046ef4153c6ba1ce26f0aa"
server_key_iv: "eefd1431b357d327dc40ef702eedee1c"
server_key_version: "encryptionkey@0"
I20260925 05:08:17.247766 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "15483d368c534a28b0f2904cae335788"
format_stamp: "Formatted at 2026-09-25 05:08:17 on dist-test-slave-wpdm"
server_key: "28e92e967a046ef4153c6ba1ce26f0aa"
server_key_iv: "eefd1431b357d327dc40ef702eedee1c"
server_key_version: "encryptionkey@0"
I20260925 05:08:17.252601 19202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260925 05:08:17.255900 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:17.257205 19202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:17.257390 19202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "15483d368c534a28b0f2904cae335788"
format_stamp: "Formatted at 2026-09-25 05:08:17 on dist-test-slave-wpdm"
server_key: "28e92e967a046ef4153c6ba1ce26f0aa"
server_key_iv: "eefd1431b357d327dc40ef702eedee1c"
server_key_version: "encryptionkey@0"
I20260925 05:08:17.257613 19202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:17.263105 19202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:17.263863 19202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:17.264127 19202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:17.265048 19202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:17.266392 19202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:17.266510 19202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:17.266645 19202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:17.266732 19202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:17.299371 19202 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:33079
I20260925 05:08:17.299422 19328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:33079 every 8 connection(s)
I20260925 05:08:17.300766 19202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:08:17.308912 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.303765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:47857 (local address 127.8.215.62:42743)
0925 05:08:17.303956 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:17.303960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:17.305205 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 05:08:17.306799 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:17.306803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:17.306853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:17.306938 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:17.308088 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:17.308092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:17.308095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:17.308110 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:17.308141 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:17.308152 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:17.308253 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:17.308521 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 05:08:17.308568 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:08:17.309652 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 19202
I20260925 05:08:17.309836 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:08:17.310245 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.304063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42743 (local address 127.8.215.3:47857)
0925 05:08:17.305049 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:17.305097 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:17.306551 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:17.307139 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:17.307147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:17.307233 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:17.307818 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:08:17.307821 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:17.307856 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:08:17.307934 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:17.308305 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:17.308314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:17.308321 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:08:17.308580 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 05:08:17.308868 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":275,"threads_started":1}
I20260925 05:08:17.310417 9052 external_mini_cluster.cc:1428] Setting key 02c304bc502e44de3f16418be40cda80
I20260925 05:08:17.311663 19329 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42743
I20260925 05:08:17.311964 19329 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:17.312633 19329 heartbeater.cc:511] Master 127.8.215.62:42743 requested a full tablet report, sending...
I20260925 05:08:17.314128 18880 ts_manager.cc:194] Registered new tserver with Master: 15483d368c534a28b0f2904cae335788 (127.8.215.3:33079)
I20260925 05:08:17.325665 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:08:17.331561 19070 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:17.330033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.8.215.62:42743)
0925 05:08:17.330186 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:17.330190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:17.330753 (+ 563us) server_negotiation.cc:407] Connection header received
0925 05:08:17.330885 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:17.330889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:17.330938 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:17.331017 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:17.331310 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58622: BlockingRecv error: recv got EOF from 127.0.0.1:58622 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:08:17.332470 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18941
I20260925 05:08:17.335942 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.333437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.8.215.1:38433)
0925 05:08:17.333978 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:17.334006 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:17.334040 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:08:17.334109 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:17.334115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:17.334361 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:17.334482 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:17.335330 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:17.335336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:17.335348 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:17.335359 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:17.335400 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:17.335416 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:17.335515 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:17.335641 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:08:17.335722 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":290,"threads_started":1}
I20260925 05:08:17.337116 19063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:17.418371 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18941
I20260925 05:08:17.434908 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 19072
I20260925 05:08:17.438196 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.435769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.8.215.2:40925)
0925 05:08:17.436276 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:17.436284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:17.436311 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:08:17.436372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:17.436378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:17.436599 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:17.436800 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:17.437600 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:17.437607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:17.437619 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:17.437631 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:17.437705 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:17.437723 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:17.437824 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:17.437900 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:08:17.437979 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20260925 05:08:17.439369 19194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:17.518559 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 19072
I20260925 05:08:17.534957 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 19202
I20260925 05:08:17.539063 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.536043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.8.215.3:33079)
0925 05:08:17.536686 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:17.536693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:17.536774 (+ 81us) server_negotiation.cc:407] Connection header received
0925 05:08:17.536845 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:17.536851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:17.537060 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:17.537222 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:17.538332 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:17.538337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:17.538350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:08:17.538364 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:17.538402 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:08:17.538415 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:17.538593 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:17.538664 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 05:08:17.538790 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":345,"threads_started":1}
I20260925 05:08:17.540246 19324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:17.620397 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 19202
I20260925 05:08:17.636581 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 18850
I20260925 05:08:17.637760 18911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:17.737952 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 18850
2026-09-25T05:08:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2310 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:08:17 dist-test-slave-wpdm krb5kdc[19347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:08:17 dist-test-slave-wpdm krb5kdc[19347](info): set up 2 sockets
Sep 25 05:08:17 dist-test-slave-wpdm krb5kdc[19347](info): commencing operation
krb5kdc: starting...
W20260925 05:08:18.850157 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:08:19 dist-test-slave-wpdm krb5kdc[19347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:08:19.092543 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:38771
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:38771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:19.414834 19360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:19.415226 19360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:19.415390 19360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:19.422608 19360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:19.422750 19360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:19.422864 19360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:19.422958 19360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:19.423046 19360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:19.432763 19360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:38771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:38771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:19.434091 19360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:19.435457 19360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:19.447227 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:19.447141 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:19.447171 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:19.447872 19360 server_base.cc:1034] running on GCE node
I20260925 05:08:19.448572 19360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:19.449878 19360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:19.451072 19360 hybrid_clock.cc:648] HybridClock initialized: now 1790312899451029 us; error 39 us; skew 500 ppm
I20260925 05:08:19.451409 19360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:08:19 dist-test-slave-wpdm krb5kdc[19347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:19.454392 19360 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:08:19.455730 19360 webserver.cc:460] Webserver started at http://127.8.215.62:39871/ using document root <none> and password file <none>
I20260925 05:08:19.456390 19360 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:19.456533 19360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:19.456897 19360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:19.459726 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "d41891c4b97f460aaf129c9c78ee2310"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-wpdm"
server_key: "1267fafc80ca559bd95afeb047721310"
server_key_iv: "6164ccd060df72edf853875572a92cd7"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.460457 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "d41891c4b97f460aaf129c9c78ee2310"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-wpdm"
server_key: "1267fafc80ca559bd95afeb047721310"
server_key_iv: "6164ccd060df72edf853875572a92cd7"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.465353 19360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:08:19.468611 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:19.469899 19360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:08:19.470073 19360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "d41891c4b97f460aaf129c9c78ee2310"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-wpdm"
server_key: "1267fafc80ca559bd95afeb047721310"
server_key_iv: "6164ccd060df72edf853875572a92cd7"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.470286 19360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:19.489552 19360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:19.493829 19360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:19.494112 19360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:19.511745 19360 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:38771
I20260925 05:08:19.511775 19426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:38771 every 8 connection(s)
I20260925 05:08:19.513134 19360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:19.515203 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 19360
I20260925 05:08:19.515446 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:08:19.516237 9052 external_mini_cluster.cc:1428] Setting key 384dd0d6aae07fb1f370d49a6d58393a
I20260925 05:08:19.517714 19427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:08:19 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:08:19.531700 19427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310: Bootstrap starting.
I20260925 05:08:19.535069 19427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:19.536118 19427 log.cc:824] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:19.538792 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:19.518295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.8.215.62:38771)
0925 05:08:19.519076 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:19.519093 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:19.519134 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:08:19.520632 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:19.520659 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:19.521111 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:19.521440 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:19.523234 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.524568 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.526375 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.527021 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.527148 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:19.531692 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:19.531700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:19.531715 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:19.531753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:19.534724 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:19.535473 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:19.535480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:19.535487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:19.535655 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:19.536023 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:19.536028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:19.536031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:19.536570 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:19.536953 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:19.537443 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 05:08:19.537845 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":381,"threads_started":1}
I20260925 05:08:19.539089 19427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310: No bootstrap required, opened a new log
I20260925 05:08:19.546519 19427 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41891c4b97f460aaf129c9c78ee2310" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38771 } }
I20260925 05:08:19.546919 19427 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:19.547029 19427 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41891c4b97f460aaf129c9c78ee2310, State: Initialized, Role: FOLLOWER
I20260925 05:08:19.547667 19427 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41891c4b97f460aaf129c9c78ee2310" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38771 } }
I20260925 05:08:19.547923 19427 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:19.548063 19427 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:19.548231 19427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:19.550407 19427 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41891c4b97f460aaf129c9c78ee2310" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38771 } }
I20260925 05:08:19.550879 19427 leader_election.cc:302] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41891c4b97f460aaf129c9c78ee2310; no voters:
I20260925 05:08:19.551455 19427 leader_election.cc:288] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:08:19.551585 19432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:19.552270 19432 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [term 1 LEADER]: Becoming Leader. State: Replica: d41891c4b97f460aaf129c9c78ee2310, State: Running, Role: LEADER
I20260925 05:08:19.552852 19432 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41891c4b97f460aaf129c9c78ee2310" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38771 } }
I20260925 05:08:19.553639 19427 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:19.556017 19434 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41891c4b97f460aaf129c9c78ee2310. Latest consensus state: current_term: 1 leader_uuid: "d41891c4b97f460aaf129c9c78ee2310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41891c4b97f460aaf129c9c78ee2310" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38771 } } }
I20260925 05:08:19.556407 19434 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:19.556107 19433 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41891c4b97f460aaf129c9c78ee2310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41891c4b97f460aaf129c9c78ee2310" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38771 } } }
I20260925 05:08:19.556512 19433 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:19.558784 19438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:19.563753 19438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:19.572988 19438 catalog_manager.cc:1347] Generated new cluster ID: aed5147a79b64186b303e3287096a6ed
I20260925 05:08:19.573129 19438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:08:19.594489 19438 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:08:19.596204 19438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:19.604650 19438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310: Generated new TSK 0
I20260925 05:08:19.605726 19438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:08:19.674825 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38771
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:20.001893 19453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:20.002179 19453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:20.002315 19453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:20.009379 19453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:20.009501 19453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:20.009692 19453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:08:20.019011 19453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:20.020260 19453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:20.021535 19453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:20.034622 19458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.034677 19459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.034667 19461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:20.035425 19453 server_base.cc:1034] running on GCE node
I20260925 05:08:20.036007 19453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:20.036998 19453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:20.038247 19453 hybrid_clock.cc:648] HybridClock initialized: now 1790312900038191 us; error 47 us; skew 500 ppm
I20260925 05:08:20.038691 19453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:08:20 dist-test-slave-wpdm krb5kdc[19347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:20.041841 19453 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:08:20.043406 19453 webserver.cc:460] Webserver started at http://127.8.215.1:46773/ using document root <none> and password file <none>
I20260925 05:08:20.044070 19453 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:20.044222 19453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:20.044517 19453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:20.047376 19453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "08676dc0833a42daa5c8ed4ccec264d0"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-wpdm"
server_key: "c4b5c4d6e086b575cbabfcf48094c432"
server_key_iv: "2a961371ee7615bab5bbc9fc68e11ec1"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.048134 19453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "08676dc0833a42daa5c8ed4ccec264d0"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-wpdm"
server_key: "c4b5c4d6e086b575cbabfcf48094c432"
server_key_iv: "2a961371ee7615bab5bbc9fc68e11ec1"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.053017 19453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 05:08:20.056666 19468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.058089 19453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:20.058248 19453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "08676dc0833a42daa5c8ed4ccec264d0"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-wpdm"
server_key: "c4b5c4d6e086b575cbabfcf48094c432"
server_key_iv: "2a961371ee7615bab5bbc9fc68e11ec1"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.058413 19453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:20.084262 19453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:20.088629 19453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:20.088938 19453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:20.089879 19453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:20.091162 19453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:20.091229 19453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.091399 19453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:20.091455 19453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.123508 19453 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:38739
I20260925 05:08:20.123708 19580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:38739 every 8 connection(s)
I20260925 05:08:20.124807 19453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:20.130209 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 19453
I20260925 05:08:20.130445 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:08:20.131234 9052 external_mini_cluster.cc:1428] Setting key ee9feefccaac9f5fe181d6deaabeee18
Sep 25 05:08:20 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:08:20.146437 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.127278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:53497 (local address 127.8.215.62:38771)
0925 05:08:20.127684 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.127688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.128541 (+ 853us) server_negotiation.cc:407] Connection header received
0925 05:08:20.130660 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.130664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.130728 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.130820 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:20.132698 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.133643 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.136194 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.136611 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.136710 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.140901 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:20.140905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:20.140909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:20.140938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:20.143465 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.144205 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.144211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.144214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.144332 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.144764 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.144769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.144774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.145031 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:20.145186 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.145850 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.146056 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":222,"threads_started":1}
I20260925 05:08:20.147419 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.127712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38771 (local address 127.8.215.1:53497)
0925 05:08:20.128385 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.128421 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.130340 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.131106 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.131120 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.131861 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:20.132522 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.132559 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.133808 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.133816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.136030 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.136047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.136852 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.136861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.137175 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.138003 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:20.138036 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:20.140644 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:20.143640 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.143649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.143665 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.144082 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.144438 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.144444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.144448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.144634 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.145163 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:20.145171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:20.145490 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.145760 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.146294 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":267,"threads_started":1}
I20260925 05:08:20.149048 19581 heartbeater.cc:347] Connected to a master server at 127.8.215.62:38771
I20260925 05:08:20.149394 19581 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:20.150236 19581 heartbeater.cc:511] Master 127.8.215.62:38771 requested a full tablet report, sending...
I20260925 05:08:20.152668 19392 ts_manager.cc:194] Registered new tserver with Master: 08676dc0833a42daa5c8ed4ccec264d0 (127.8.215.1:38739)
I20260925 05:08:20.154953 19392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:53497
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:08:20.187469 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38771
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:20.506155 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:20.506424 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:20.506620 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:20.513775 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:20.513876 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:20.514060 19587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:08:20.523561 19587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:20.524837 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:20.526080 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:20.539204 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.539204 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.539136 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:20.539681 19587 server_base.cc:1034] running on GCE node
I20260925 05:08:20.540297 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:20.541244 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:20.542464 19587 hybrid_clock.cc:648] HybridClock initialized: now 1790312900542429 us; error 51 us; skew 500 ppm
I20260925 05:08:20.542776 19587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:08:20 dist-test-slave-wpdm krb5kdc[19347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:20.545866 19587 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:08:20.547298 19587 webserver.cc:460] Webserver started at http://127.8.215.2:44077/ using document root <none> and password file <none>
I20260925 05:08:20.547951 19587 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:20.548126 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:20.548418 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:20.551225 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "0993b77aed0444828cf6df3459e5e2c1"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-wpdm"
server_key: "0fa56d09c556f0e2861d1bbe69bbf6fd"
server_key_iv: "30af954e43d8d46cc1e876d53a4c186b"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.551899 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "0993b77aed0444828cf6df3459e5e2c1"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-wpdm"
server_key: "0fa56d09c556f0e2861d1bbe69bbf6fd"
server_key_iv: "30af954e43d8d46cc1e876d53a4c186b"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.556977 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:08:20.560297 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.561793 19587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:20.561941 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "0993b77aed0444828cf6df3459e5e2c1"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-wpdm"
server_key: "0fa56d09c556f0e2861d1bbe69bbf6fd"
server_key_iv: "30af954e43d8d46cc1e876d53a4c186b"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.562150 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:20.578529 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:20.582401 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:20.582680 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:20.583673 19587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:20.584975 19587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:20.585044 19587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.585196 19587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:20.585294 19587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.617614 19587 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:33301
I20260925 05:08:20.617729 19714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:33301 every 8 connection(s)
I20260925 05:08:20.618935 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:08:20.619591 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 19587
I20260925 05:08:20.619854 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:08:20.620502 9052 external_mini_cluster.cc:1428] Setting key 258f4723ef7cdac8ac3731944391dcd7
Sep 25 05:08:20 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:08:20.642067 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.621728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:46629 (local address 127.8.215.62:38771)
0925 05:08:20.621933 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.621939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.623250 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 05:08:20.626228 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.626232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.626347 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.626445 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:20.628598 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.629520 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.632098 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.632520 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.632656 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.636376 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:20.636381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:20.636385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:20.636415 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:20.638987 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.639728 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.639734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.639738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.639868 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.640302 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.640307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.640310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.640572 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:20.640727 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.641494 (+ 767us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.641723 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:08:20.643170 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.622218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38771 (local address 127.8.215.2:46629)
0925 05:08:20.623062 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.623125 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.625883 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.626724 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.626738 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.627544 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:20.628384 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.628415 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.629688 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.629697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.631936 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.631955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.632793 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.632802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.633023 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.633700 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:20.633721 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:20.636136 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:20.639139 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.639150 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.639168 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.639595 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.639982 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.639987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.639991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.640181 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.640704 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:20.640713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:20.641123 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.641398 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.641966 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":327,"threads_started":1}
I20260925 05:08:20.644754 19715 heartbeater.cc:347] Connected to a master server at 127.8.215.62:38771
I20260925 05:08:20.645066 19715 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:20.645797 19715 heartbeater.cc:511] Master 127.8.215.62:38771 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
I20260925 05:08:20.647423 19392 ts_manager.cc:194] Registered new tserver with Master: 0993b77aed0444828cf6df3459e5e2c1 (127.8.215.2:33301)
I20260925 05:08:20.648415 19392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:46629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:08:20.680213 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38771
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:21.013302 19720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:21.013607 19720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:21.013775 19720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:21.021325 19720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:21.021451 19720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:21.021658 19720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:08:21.031680 19720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:08:21.033097 19720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:21.034341 19720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:21.047410 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:21.047523 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:21.047806 19728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:21.049224 19720 server_base.cc:1034] running on GCE node
I20260925 05:08:21.049865 19720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:21.050849 19720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:21.052073 19720 hybrid_clock.cc:648] HybridClock initialized: now 1790312901052019 us; error 36 us; skew 500 ppm
I20260925 05:08:21.052415 19720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:08:21 dist-test-slave-wpdm krb5kdc[19347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312901, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:21.055691 19720 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:08:21.057150 19720 webserver.cc:460] Webserver started at http://127.8.215.3:46105/ using document root <none> and password file <none>
I20260925 05:08:21.057844 19720 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:21.057941 19720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:21.058210 19720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:21.061198 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "6a3af1bb14bc4cc29efba841c59a16eb"
format_stamp: "Formatted at 2026-09-25 05:08:21 on dist-test-slave-wpdm"
server_key: "5781ff9dd30457bd200e928a3040df9d"
server_key_iv: "9b6ee47efd1857897aad255c45e2278a"
server_key_version: "encryptionkey@0"
I20260925 05:08:21.062076 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "6a3af1bb14bc4cc29efba841c59a16eb"
format_stamp: "Formatted at 2026-09-25 05:08:21 on dist-test-slave-wpdm"
server_key: "5781ff9dd30457bd200e928a3040df9d"
server_key_iv: "9b6ee47efd1857897aad255c45e2278a"
server_key_version: "encryptionkey@0"
I20260925 05:08:21.067539 19720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:08:21.070879 19735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:21.072204 19720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:08:21.072403 19720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "6a3af1bb14bc4cc29efba841c59a16eb"
format_stamp: "Formatted at 2026-09-25 05:08:21 on dist-test-slave-wpdm"
server_key: "5781ff9dd30457bd200e928a3040df9d"
server_key_iv: "9b6ee47efd1857897aad255c45e2278a"
server_key_version: "encryptionkey@0"
I20260925 05:08:21.072590 19720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:21.085822 19720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:21.089998 19720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:21.090307 19720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:21.091270 19720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:21.092566 19720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:21.092667 19720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:21.092835 19720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:21.092912 19720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:21.126459 19720 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39197
I20260925 05:08:21.126494 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39197 every 8 connection(s)
I20260925 05:08:21.127779 19720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:08:21.133705 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 19720
I20260925 05:08:21.133952 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:08:21.134578 9052 external_mini_cluster.cc:1428] Setting key 7dabd5b7f92e7d970a24b8a01a6af5b7
Sep 25 05:08:21 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312901, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:08:21.147701 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.130531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:53557 (local address 127.8.215.62:38771)
0925 05:08:21.130699 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.130703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.131808 (+ 1105us) server_negotiation.cc:407] Connection header received
0925 05:08:21.133668 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.133674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.133774 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.133886 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:21.135944 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.136880 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.138671 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.139047 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.139142 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.142499 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:21.142504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:21.142507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:21.142537 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:21.145204 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.145857 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.145861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.145863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.145977 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.146338 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.146341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.146343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.146579 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:21.146684 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.147299 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.147447 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:08:21.148691 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.130965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38771 (local address 127.8.215.3:53557)
0925 05:08:21.131656 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:21.131691 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:21.133396 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:21.134141 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:21.134149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:21.134988 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:21.135746 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.135774 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.137023 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.137029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.138541 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.138557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.139220 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.139225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.139426 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.140102 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:21.140126 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:21.142305 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:21.145323 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:21.145331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:21.145344 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:21.145741 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:21.146068 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:21.146071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:21.146073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:21.146248 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:21.146679 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:21.146684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:21.146987 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 05:08:21.147213 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 05:08:21.147577 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":288,"threads_started":1}
I20260925 05:08:21.150111 19848 heartbeater.cc:347] Connected to a master server at 127.8.215.62:38771
I20260925 05:08:21.150444 19848 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:21.151098 19848 heartbeater.cc:511] Master 127.8.215.62:38771 requested a full tablet report, sending...
I20260925 05:08:21.152365 19392 ts_manager.cc:194] Registered new tserver with Master: 6a3af1bb14bc4cc29efba841c59a16eb (127.8.215.3:39197)
I20260925 05:08:21.153185 19392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:53557
I20260925 05:08:21.158211 19581 heartbeater.cc:503] Master 127.8.215.62:38771 was elected leader, sending a full tablet report...
I20260925 05:08:21.162401 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:08:21.179466 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.167025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.8.215.62:38771)
0925 05:08:21.167236 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.167244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.167653 (+ 409us) server_negotiation.cc:407] Connection header received
0925 05:08:21.167784 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.167788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.167847 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.167927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:21.169470 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.170160 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.171317 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.171743 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.171833 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.174167 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:21.174172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:21.174175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:21.174206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:21.177020 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.177739 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.177742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.177744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.177864 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.178179 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.178182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.178184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.178530 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:21.178710 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.178982 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.179131 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:08:21.183741 19392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:08:21.190601 19392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:08:21.214617 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.203020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.8.215.2:33301)
0925 05:08:21.205206 (+ 2186us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.205218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.205596 (+ 378us) server_negotiation.cc:407] Connection header received
0925 05:08:21.205791 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.205802 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.206073 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.206278 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:21.207746 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.209354 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.212399 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.213409 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.213564 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.213767 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.214108 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.214283 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2032,"thread_start_us":1937,"threads_started":1}
I20260925 05:08:21.215277 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.202648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.8.215.1:38739)
0925 05:08:21.206188 (+ 3540us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.206198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.206224 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:08:21.206319 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.206328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.206549 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.207473 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:21.207709 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.209356 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.212266 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.213675 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.213790 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.214010 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.214798 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.214952 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3364,"thread_start_us":3215,"threads_started":1}
I20260925 05:08:21.215587 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.206341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.8.215.3:39197)
0925 05:08:21.207090 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.207098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.207128 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:08:21.207219 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.207226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.207430 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.208104 (+ 674us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:21.208314 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.209386 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.212106 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.213551 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.214809 (+ 1258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.215013 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.215080 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.215249 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":454,"threads_started":1}
I20260925 05:08:21.215719 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.205754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39197 (local address 127.0.0.1:45668)
0925 05:08:21.206408 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:21.206427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:21.206543 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:21.207630 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:21.207634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:21.207659 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:21.207966 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.207974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.209568 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.209575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.211954 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.211966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.213785 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.213791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.213994 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.214028 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:08:21.214102 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:08:21.214253 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":196,"thread_start_us":829,"threads_started":1}
I20260925 05:08:21.216082 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.202558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:33301 (local address 127.0.0.1:49166)
0925 05:08:21.205484 (+ 2926us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:21.205498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:21.205634 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:21.206520 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:21.206527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:21.206555 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:21.207543 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.207556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.209534 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.209541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.212255 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.212266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.213600 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.213605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.213870 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.213917 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:08:21.214056 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 05:08:21.214248 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2832,"mutex_wait_us":216,"thread_start_us":2722,"threads_started":1}
I20260925 05:08:21.216289 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.202445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38739 (local address 127.0.0.1:55576)
0925 05:08:21.203162 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:21.203211 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:21.203373 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:21.206756 (+ 3383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:21.206762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:21.206785 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:21.207237 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.207254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.209496 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.209504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.212102 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.212122 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.214150 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.214157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.214326 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.214354 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:08:21.214442 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:08:21.214551 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":126,"thread_start_us":748,"threads_started":1}
I20260925 05:08:21.219291 19516 tablet_service.cc:1467] Processing CreateTablet for tablet ba6156d0fbcb4af688199aceead3e1b9 (DEFAULT_TABLE table=test-table [id=e915b75aba4c4c6ca1c1b8421a735a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:21.219285 19650 tablet_service.cc:1467] Processing CreateTablet for tablet ba6156d0fbcb4af688199aceead3e1b9 (DEFAULT_TABLE table=test-table [id=e915b75aba4c4c6ca1c1b8421a735a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:21.220652 19516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6156d0fbcb4af688199aceead3e1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:21.220657 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6156d0fbcb4af688199aceead3e1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:21.222424 19783 tablet_service.cc:1467] Processing CreateTablet for tablet ba6156d0fbcb4af688199aceead3e1b9 (DEFAULT_TABLE table=test-table [id=e915b75aba4c4c6ca1c1b8421a735a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:21.223665 19783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6156d0fbcb4af688199aceead3e1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:21.231022 19866 tablet_bootstrap.cc:492] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Bootstrap starting.
I20260925 05:08:21.234915 19866 tablet_bootstrap.cc:654] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:21.235062 19867 tablet_bootstrap.cc:492] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Bootstrap starting.
I20260925 05:08:21.236325 19866 log.cc:824] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:21.237372 19868 tablet_bootstrap.cc:492] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Bootstrap starting.
I20260925 05:08:21.238653 19867 tablet_bootstrap.cc:654] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:21.239856 19867 log.cc:824] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:21.240392 19868 tablet_bootstrap.cc:654] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:21.240795 19866 tablet_bootstrap.cc:492] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: No bootstrap required, opened a new log
I20260925 05:08:21.241086 19866 ts_tablet_manager.cc:1397] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260925 05:08:21.241652 19868 log.cc:824] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:21.243043 19867 tablet_bootstrap.cc:492] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: No bootstrap required, opened a new log
I20260925 05:08:21.243345 19867 ts_tablet_manager.cc:1397] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 05:08:21.248443 19866 raft_consensus.cc:348] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.248869 19866 raft_consensus.cc:374] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:21.248966 19866 raft_consensus.cc:729] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0993b77aed0444828cf6df3459e5e2c1, State: Initialized, Role: FOLLOWER
I20260925 05:08:21.249560 19866 consensus_queue.cc:260] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.250695 19715 heartbeater.cc:503] Master 127.8.215.62:38771 was elected leader, sending a full tablet report...
I20260925 05:08:21.251070 19868 tablet_bootstrap.cc:492] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: No bootstrap required, opened a new log
I20260925 05:08:21.251219 19866 ts_tablet_manager.cc:1428] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260925 05:08:21.251325 19868 ts_tablet_manager.cc:1397] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260925 05:08:21.255347 19867 raft_consensus.cc:348] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.255777 19867 raft_consensus.cc:374] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:21.255900 19867 raft_consensus.cc:729] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08676dc0833a42daa5c8ed4ccec264d0, State: Initialized, Role: FOLLOWER
I20260925 05:08:21.256522 19867 consensus_queue.cc:260] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.258311 19867 ts_tablet_manager.cc:1428] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 05:08:21.260093 19868 raft_consensus.cc:348] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.260497 19868 raft_consensus.cc:374] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:21.260603 19868 raft_consensus.cc:729] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3af1bb14bc4cc29efba841c59a16eb, State: Initialized, Role: FOLLOWER
I20260925 05:08:21.261258 19868 consensus_queue.cc:260] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.262323 19848 heartbeater.cc:503] Master 127.8.215.62:38771 was elected leader, sending a full tablet report...
I20260925 05:08:21.262900 19868 ts_tablet_manager.cc:1428] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260925 05:08:21.371183 19716 tablet.cc:2369] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:08:21.377008 19582 tablet.cc:2369] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:08:21.379462 19849 tablet.cc:2369] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:08:21.599822 19873 raft_consensus.cc:482] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:08:21.600153 19873 raft_consensus.cc:504] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.601809 19873 leader_election.cc:288] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0993b77aed0444828cf6df3459e5e2c1 (127.8.215.2:33301), 6a3af1bb14bc4cc29efba841c59a16eb (127.8.215.3:39197)
I20260925 05:08:21.608326 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.602200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:56841 (local address 127.8.215.2:33301)
0925 05:08:21.602416 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.602425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.602983 (+ 558us) server_negotiation.cc:407] Connection header received
0925 05:08:21.603238 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.603247 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.603349 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.603453 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:21.604104 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.604894 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.606669 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.607422 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.607581 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.607681 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.607952 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.608074 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:08:21.608518 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.602273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:33301 (local address 127.8.215.1:56841)
0925 05:08:21.602863 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:21.602879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:21.603059 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:21.603508 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:21.603514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:21.603550 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:21.603963 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.603975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.605024 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.605029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.606520 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.606531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.607634 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.607640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.607801 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.607824 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:08:21.607928 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:08:21.608054 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":335,"threads_started":1}
I20260925 05:08:21.609311 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.603147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:41999 (local address 127.8.215.3:39197)
0925 05:08:21.603335 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.603343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.603857 (+ 514us) server_negotiation.cc:407] Connection header received
0925 05:08:21.604017 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.604021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.604085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.604226 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:21.604824 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.605548 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.607293 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.608109 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.608282 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.608398 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.608884 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.609027 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:08:21.609346 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.603016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39197 (local address 127.8.215.1:41999)
0925 05:08:21.603713 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:21.603734 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:21.603875 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:21.604239 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:21.604245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:21.604271 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:21.604623 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.604634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.605700 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.605708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.607132 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:21.607144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.608353 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.608360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:21.608647 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.608693 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 05:08:21.608860 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 05:08:21.608986 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":321,"threads_started":1}
I20260925 05:08:21.609581 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6156d0fbcb4af688199aceead3e1b9" candidate_uuid: "08676dc0833a42daa5c8ed4ccec264d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0993b77aed0444828cf6df3459e5e2c1" is_pre_election: true
I20260925 05:08:21.610116 19670 raft_consensus.cc:2454] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08676dc0833a42daa5c8ed4ccec264d0 in term 0.
I20260925 05:08:21.610538 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6156d0fbcb4af688199aceead3e1b9" candidate_uuid: "08676dc0833a42daa5c8ed4ccec264d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" is_pre_election: true
I20260925 05:08:21.610852 19470 leader_election.cc:302] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08676dc0833a42daa5c8ed4ccec264d0, 0993b77aed0444828cf6df3459e5e2c1; no voters:
I20260925 05:08:21.610976 19803 raft_consensus.cc:2454] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08676dc0833a42daa5c8ed4ccec264d0 in term 0.
I20260925 05:08:21.611264 19873 raft_consensus.cc:2781] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:08:21.611431 19873 raft_consensus.cc:482] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:08:21.611546 19873 raft_consensus.cc:3037] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:21.613782 19873 raft_consensus.cc:504] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.614274 19873 leader_election.cc:288] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [CANDIDATE]: Term 1 election: Requested vote from peers 0993b77aed0444828cf6df3459e5e2c1 (127.8.215.2:33301), 6a3af1bb14bc4cc29efba841c59a16eb (127.8.215.3:39197)
I20260925 05:08:21.614759 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6156d0fbcb4af688199aceead3e1b9" candidate_uuid: "08676dc0833a42daa5c8ed4ccec264d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0993b77aed0444828cf6df3459e5e2c1"
I20260925 05:08:21.614820 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6156d0fbcb4af688199aceead3e1b9" candidate_uuid: "08676dc0833a42daa5c8ed4ccec264d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3af1bb14bc4cc29efba841c59a16eb"
I20260925 05:08:21.614938 19670 raft_consensus.cc:3037] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:21.615032 19803 raft_consensus.cc:3037] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:21.617305 19670 raft_consensus.cc:2454] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08676dc0833a42daa5c8ed4ccec264d0 in term 1.
I20260925 05:08:21.617445 19803 raft_consensus.cc:2454] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08676dc0833a42daa5c8ed4ccec264d0 in term 1.
I20260925 05:08:21.617733 19470 leader_election.cc:302] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08676dc0833a42daa5c8ed4ccec264d0, 0993b77aed0444828cf6df3459e5e2c1; no voters:
I20260925 05:08:21.618033 19873 raft_consensus.cc:2781] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:21.618490 19873 raft_consensus.cc:686] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 1 LEADER]: Becoming Leader. State: Replica: 08676dc0833a42daa5c8ed4ccec264d0, State: Running, Role: LEADER
I20260925 05:08:21.619014 19873 consensus_queue.cc:237] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } }
I20260925 05:08:21.623821 19392 catalog_manager.cc:5495] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08676dc0833a42daa5c8ed4ccec264d0 (127.8.215.1). New cstate: current_term: 1 leader_uuid: "08676dc0833a42daa5c8ed4ccec264d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08676dc0833a42daa5c8ed4ccec264d0" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 } health_report { overall_health: UNKNOWN } } }
I20260925 05:08:21.667114 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.661589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.8.215.1:38739)
0925 05:08:21.661856 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.661862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.661882 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:08:21.662119 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.662124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.662201 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.662377 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:21.662761 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.663528 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.664905 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.665315 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.665461 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.666028 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:21.666537 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:08:21.666726 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.666775 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.666853 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:08:21.675489 19803 raft_consensus.cc:1261] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 1 FOLLOWER]: Refusing update from remote peer 08676dc0833a42daa5c8ed4ccec264d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:21.675489 19670 raft_consensus.cc:1261] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 1 FOLLOWER]: Refusing update from remote peer 08676dc0833a42daa5c8ed4ccec264d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:21.676537 19873 consensus_queue.cc:1035] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0993b77aed0444828cf6df3459e5e2c1" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:21.676873 19877 consensus_queue.cc:1035] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3af1bb14bc4cc29efba841c59a16eb" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:21.702534 19392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:21.702751 19392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:21.706132 19392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41891c4b97f460aaf129c9c78ee2310: Sending DeleteTablet for 3 replicas of tablet ba6156d0fbcb4af688199aceead3e1b9
I20260925 05:08:21.707295 19516 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6156d0fbcb4af688199aceead3e1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:21 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55576
I20260925 05:08:21.707357 19783 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6156d0fbcb4af688199aceead3e1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:21 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:45668
I20260925 05:08:21.707536 19650 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6156d0fbcb4af688199aceead3e1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:21 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:49166
I20260925 05:08:21.709661 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 19453
I20260925 05:08:21.718357 19892 tablet_replica.cc:330] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: stopping tablet replica
I20260925 05:08:21.718997 19892 raft_consensus.cc:2229] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:21.719655 19892 raft_consensus.cc:2258] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:21.723829 19890 tablet_replica.cc:330] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: stopping tablet replica
I20260925 05:08:21.724279 19890 raft_consensus.cc:2229] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:08:21.724512 19892 ts_tablet_manager.cc:1905] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:21.724946 19890 raft_consensus.cc:2258] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:21.725584 19891 tablet_replica.cc:330] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: stopping tablet replica
I20260925 05:08:21.725966 19891 raft_consensus.cc:2229] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:21.726457 19891 raft_consensus.cc:2258] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:21.728528 19891 ts_tablet_manager.cc:1905] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:21.731758 19892 ts_tablet_manager.cc:1918] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:21.731940 19892 log.cc:1196] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/ba6156d0fbcb4af688199aceead3e1b9
I20260925 05:08:21.732477 19892 ts_tablet_manager.cc:1939] T ba6156d0fbcb4af688199aceead3e1b9 P 0993b77aed0444828cf6df3459e5e2c1: Deleting consensus metadata
I20260925 05:08:21.734053 19890 ts_tablet_manager.cc:1905] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:21.734366 19377 catalog_manager.cc:4841] TS 0993b77aed0444828cf6df3459e5e2c1 (127.8.215.2:33301): tablet ba6156d0fbcb4af688199aceead3e1b9 (table test-table [id=e915b75aba4c4c6ca1c1b8421a735a66]) successfully deleted
I20260925 05:08:21.735226 19891 ts_tablet_manager.cc:1918] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:21.735390 19891 log.cc:1196] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/ba6156d0fbcb4af688199aceead3e1b9
I20260925 05:08:21.735810 19891 ts_tablet_manager.cc:1939] T ba6156d0fbcb4af688199aceead3e1b9 P 6a3af1bb14bc4cc29efba841c59a16eb: Deleting consensus metadata
I20260925 05:08:21.737383 19377 catalog_manager.cc:4841] TS 6a3af1bb14bc4cc29efba841c59a16eb (127.8.215.3:39197): tablet ba6156d0fbcb4af688199aceead3e1b9 (table test-table [id=e915b75aba4c4c6ca1c1b8421a735a66]) successfully deleted
Sep 25 05:08:21 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:08:21.743911 19890 ts_tablet_manager.cc:1918] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:21.744092 19890 log.cc:1196] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/ba6156d0fbcb4af688199aceead3e1b9
I20260925 05:08:21.744629 19890 ts_tablet_manager.cc:1939] T ba6156d0fbcb4af688199aceead3e1b9 P 08676dc0833a42daa5c8ed4ccec264d0: Deleting consensus metadata
I20260925 05:08:21.746508 19378 catalog_manager.cc:4841] TS 08676dc0833a42daa5c8ed4ccec264d0 (127.8.215.1:38739): tablet ba6156d0fbcb4af688199aceead3e1b9 (table test-table [id=e915b75aba4c4c6ca1c1b8421a735a66]) successfully deleted
I20260925 05:08:21.746773 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.711066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.8.215.1:38739)
0925 05:08:21.711304 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.711313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.723782 (+ 12469us) server_negotiation.cc:407] Connection header received
0925 05:08:21.724367 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.724373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.724456 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.724537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:21.726693 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.727981 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.729983 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.730683 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.730789 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.738830 (+ 8041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:21.738843 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:21.738863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:21.738928 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:21.743393 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.744362 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.744374 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.744381 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.744581 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.745124 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.745129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.745135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.745578 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:21.745881 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.746242 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.746463 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:08:21.747905 19576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:21.848330 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 19453
I20260925 05:08:21.867753 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 19587
Sep 25 05:08:21 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:08:21.882510 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.868567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.8.215.2:33301)
0925 05:08:21.868829 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.868835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.868855 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:08:21.868966 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.868970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.869031 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.869135 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:21.870433 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.871128 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.872287 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.872663 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.872880 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.876829 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:21.876843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:21.876859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:21.876895 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:21.879723 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.880411 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.880419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.880421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.880553 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:21.880984 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:21.880989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:21.880994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:21.881364 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:21.881664 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:21.881958 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 05:08:21.882169 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:08:21.883172 19710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:21.969978 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 19587
I20260925 05:08:21.987638 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 19720
Sep 25 05:08:21 dist-test-slave-wpdm krb5kdc[19347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:08:22.002246 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:21.988446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.8.215.3:39197)
0925 05:08:21.988633 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:21.988641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:21.988707 (+ 66us) server_negotiation.cc:407] Connection header received
0925 05:08:21.988903 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:21.988907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:21.988981 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:21.989060 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:21.990296 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.990986 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.992171 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:21.992540 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:21.992629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:21.996880 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:21.996895 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:21.996911 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:21.996944 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:21.999652 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:22.000267 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:22.000277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:22.000280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:22.000427 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:22.000811 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:22.000818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:22.000824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:22.001194 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:22.001465 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:22.001734 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:08:22.001897 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:08:22.002921 19843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:22.092981 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 19720
I20260925 05:08:22.111848 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 19360
I20260925 05:08:22.113018 19422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:08:22.218811 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 19360
2026-09-25T05:08:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4482 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:08:22 dist-test-slave-wpdm krb5kdc[19900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:08:22 dist-test-slave-wpdm krb5kdc[19900](info): set up 2 sockets
Sep 25 05:08:22 dist-test-slave-wpdm krb5kdc[19900](info): commencing operation
krb5kdc: starting...
W20260925 05:08:23.378475 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:08:23 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/rangeradmin_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/rangeradmin_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/rangerlookup_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/rangerlookup_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
I20260925 05:08:23.711190 9052 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjaqqHJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/postgres -l logfile start
2026-09-25 05:08:26.190 UTC [19930] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:08:26.190 UTC [19930] LOG: listening on IPv4 address "127.8.215.20", port 48143
2026-09-25 05:08:26.193 UTC [19930] LOG: listening on Unix socket "/tmp/.s.PGSQL.48143"
2026-09-25 05:08:26.217 UTC [19933] LOG: database system was shut down at 2026-09-25 05:08:24 UTC
2026-09-25 05:08:26.221 UTC [19930] LOG: database system is ready to accept connections
I20260925 05:08:27.197791 9052 mini_postgres.cc:96] Postgres bound to 48143
2026-09-25 05:08:27.216 UTC [19941] FATAL: database "slave" does not exist
127.8.215.20:48143 - accepting connections
I20260925 05:08:27.217751 9052 mini_ranger.cc:161] Starting Ranger...
I20260925 05:08:27.245220 9052 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:08:27.719576 9052 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:08:27.719713 9052 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-admin
2026-09-25 05:08:28,024 [I] DB FLAVOR :POSTGRES
2026-09-25 05:08:28,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:08:28,025 [I] Checking connection..
2026-09-25 05:08:28,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:08:28,264 [I] Checking connection passed.
2026-09-25 05:08:28,264 [I] --------- Verifying version history table ---------
2026-09-25 05:08:28,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:08:28,524 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:08:28,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:08:28,784 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:08:28,784 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:08:28,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:08:29.043 UTC [20005] WARNING: there is no transaction in progress
2026-09-25 05:08:29,049 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:08:29,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:08:29,319 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:08:29,319 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:08:29,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:08:29,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:08:29,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wpdm.c.gcp-upstream.internal','N') ;"
2026-09-25 05:08:30,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:08:30,273 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:08:30,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:08:30,531 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:08:30,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:08:30,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:08:31,045 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:08:31,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:08:32.009 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.025 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.041 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.056 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.074 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.094 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.154 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.162 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.177 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.187 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.198 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.211 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.221 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.228 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.235 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32.242 UTC [20113] WARNING: there is no transaction in progress
2026-09-25 05:08:32,873 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:08:32,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:08:33,124 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:08:33,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:08:33,370 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:08:33,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:08:33,606 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:08:33,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:08:33,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:08:34,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wpdm.c.gcp-upstream.internal';"
2026-09-25 05:08:34,317 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:08:34,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:08:34,564 [I] DB_PATCHES have already been applied
I20260925 05:08:34.570405 9052 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjaqqHJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:08:34.570489 9052 mini_ranger.cc:193] Using host: 127.8.215.20
I20260925 05:08:34.586891 9052 mini_ranger.cc:227] Ranger admin URL: http://127.8.215.20:59191
Sep 25, 2026 5:08:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:08:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:08:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:08:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:08:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:08:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:08:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:08:35.715184 9052 mini_ranger.cc:160] Time spent starting Ranger: real 8.497s user 0.008s sys 0.037s
Sep 25, 2026 5:08:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:08:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 5:08:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:08:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:08:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:08:57.047732 9052 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/rangerkms_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/rangerkms_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.215.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.215.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:08:57 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312937, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:08:57.230233 9052 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:08:57.252923 9052 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:08:58.167178 9052 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:08:58.167359 9052 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms
I20260925 05:08:58.167486 9052 mini_ranger_kms.cc:227] Using postgres at 127.8.215.20:48143
2026-09-25 05:08:58,604 [I] DB FLAVOR :POSTGRES
2026-09-25 05:08:58,604 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:08:58,605 [I] Checking connection
2026-09-25 05:08:58,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:08:58,843 [I] connection success
2026-09-25 05:08:58,843 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:08:58,844 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:08:58,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:08:59,083 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:08:59,083 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:08:59,084 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:08:59,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:48143/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:08:59,354 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:08:59.851236 9052 mini_ranger_kms.cc:310] Created kms service
I20260925 05:09:00.020932 9052 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:09:00.065038 9052 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:09:00 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312937, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:00.442096 9052 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:09:00.442446 9052 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:09:00.442519 9052 mini_ranger_kms.cc:240] Using host: 127.8.215.20
I20260925 05:09:00.467880 9052 mini_ranger_kms.cc:276] Ranger KMS PID: 20293
I20260925 05:09:00.468087 9052 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.215.20:41527
Sep 25, 2026 5:09:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:09:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:09:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:09:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:09:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:09:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 05:09:02 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.215.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:02.818255 9052 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.588s user 0.100s sys 0.356s
I20260925 05:09:02.818516 9052 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:09:02.818765 9052 mini_ranger_kms.cc:401] 127.8.215.20:41527/kms/v1/keys
Sep 25 05:09:03 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.215.20@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
Sep 25 05:09:03 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.215.20@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
Sep 25 05:09:06 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:09:06.086391 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43887
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43887
--ranger_config_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:41527/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:06.439302 20344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:06.439657 20344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:06.439807 20344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:06.447324 20344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:06.447440 20344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:06.447525 20344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:06.447628 20344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:06.447824 20344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:06.458722 20344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.215.20:41527/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43887
--ranger_config_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:09:06.460107 20344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:06.461504 20344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:06.476076 20350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:06.476084 20349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:06.476832 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:06.476946 20344 server_base.cc:1034] running on GCE node
I20260925 05:09:06.478039 20344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:06.480036 20344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:06.481315 20344 hybrid_clock.cc:648] HybridClock initialized: now 1790312946481248 us; error 56 us; skew 500 ppm
I20260925 05:09:06.481782 20344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:09:06 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:06.485816 20344 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:09:06.487524 20344 webserver.cc:460] Webserver started at http://127.8.215.62:39933/ using document root <none> and password file <none>
I20260925 05:09:06.488217 20344 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:06.488371 20344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:06.488690 20344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:09:06 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:06.553071 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "7e438e1ba03f4fc8867d611ea6e28629"
format_stamp: "Formatted at 2026-09-25 05:09:06 on dist-test-slave-wpdm"
server_key: "469dcc3895171e283424b1caa24b3426"
server_key_iv: "74e6a73ffcb1503d167934366aefae8b"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:06.553952 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "7e438e1ba03f4fc8867d611ea6e28629"
format_stamp: "Formatted at 2026-09-25 05:09:06 on dist-test-slave-wpdm"
server_key: "469dcc3895171e283424b1caa24b3426"
server_key_iv: "74e6a73ffcb1503d167934366aefae8b"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:06.559784 20344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:09:06.579883 20359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:06.581585 20344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:09:06.581791 20344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "7e438e1ba03f4fc8867d611ea6e28629"
format_stamp: "Formatted at 2026-09-25 05:09:06 on dist-test-slave-wpdm"
server_key: "469dcc3895171e283424b1caa24b3426"
server_key_iv: "74e6a73ffcb1503d167934366aefae8b"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:06.582024 20344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:06.607692 20344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:06.612159 20344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:06.612501 20344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:06.633556 20344 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43887
I20260925 05:09:06.633576 20410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43887 every 8 connection(s)
I20260925 05:09:06.635054 20344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:06.636159 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 20344
I20260925 05:09:06.636363 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:09:06.637188 9052 external_mini_cluster.cc:1428] Setting key 6cb7e612bf3d34021e0e9be088611e0c
I20260925 05:09:06.640995 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:06.656054 20411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629: Bootstrap starting.
Sep 25 05:09:06 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:09:06.660492 20411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:06.661772 20411 log.cc:824] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:06.666254 20411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629: No bootstrap required, opened a new log
I20260925 05:09:06.670454 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:06.639648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.8.215.62:43887)
0925 05:09:06.645397 (+ 5749us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:06.645414 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:06.646046 (+ 632us) server_negotiation.cc:407] Connection header received
0925 05:09:06.648299 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:06.648331 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:06.648953 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:06.649403 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:06.651067 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:06.653086 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:06.654963 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:06.656006 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:06.656175 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:06.661412 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:06.661423 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:06.661444 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:06.661495 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:06.665280 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:06.666117 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:06.666125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:06.666134 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:06.666335 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:06.666771 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:06.666776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:06.666779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:06.667453 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:06.667767 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:06.668520 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 05:09:06.669013 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5469,"thread_start_us":5240,"threads_started":1}
I20260925 05:09:06.675932 20411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e438e1ba03f4fc8867d611ea6e28629" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43887 } }
I20260925 05:09:06.676302 20411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:06.676397 20411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e438e1ba03f4fc8867d611ea6e28629, State: Initialized, Role: FOLLOWER
I20260925 05:09:06.677047 20411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e438e1ba03f4fc8867d611ea6e28629" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43887 } }
I20260925 05:09:06.677310 20411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:06.677438 20411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:06.677618 20411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:06.680159 20411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e438e1ba03f4fc8867d611ea6e28629" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43887 } }
I20260925 05:09:06.680691 20411 leader_election.cc:302] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e438e1ba03f4fc8867d611ea6e28629; no voters:
I20260925 05:09:06.681425 20411 leader_election.cc:288] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:06.681602 20416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:06.682245 20416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [term 1 LEADER]: Becoming Leader. State: Replica: 7e438e1ba03f4fc8867d611ea6e28629, State: Running, Role: LEADER
I20260925 05:09:06.683038 20416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e438e1ba03f4fc8867d611ea6e28629" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43887 } }
I20260925 05:09:06.685576 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e438e1ba03f4fc8867d611ea6e28629. Latest consensus state: current_term: 1 leader_uuid: "7e438e1ba03f4fc8867d611ea6e28629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e438e1ba03f4fc8867d611ea6e28629" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43887 } } }
I20260925 05:09:06.686019 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:06.686357 20418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e438e1ba03f4fc8867d611ea6e28629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e438e1ba03f4fc8867d611ea6e28629" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43887 } } }
I20260925 05:09:06.686671 20418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:06.690595 20411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:06.699697 20420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:06.700176 20411 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:09:06.713852 20420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:06.736316 20420 catalog_manager.cc:1347] Generated new cluster ID: b45c95d847d24e9aa0fb0f6f1148eda3
I20260925 05:09:06.736506 20420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:06.768610 20420 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:06.771086 20420 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:06.791117 20420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629: Generated new TSK 0
I20260925 05:09:06.792466 20420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:09:07.589856 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:41527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43887
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:09:07 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312947, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 05:09:08.052418 20456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:08.052883 20456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:08.053057 20456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:08.065104 20456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:08.065344 20456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:08.065573 20456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:09:08.081950 20456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.215.20:41527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:09:08.083669 20456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:08.085497 20456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:08.103343 20467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:08.103799 20466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:08.111649 20469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:08.112762 20456 server_base.cc:1034] running on GCE node
I20260925 05:09:08.113672 20456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:08.116405 20456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:08.117695 20456 hybrid_clock.cc:648] HybridClock initialized: now 1790312948117638 us; error 63 us; skew 500 ppm
I20260925 05:09:08.118182 20456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:09:08 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:08.122478 20456 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:09:08.131150 20456 webserver.cc:460] Webserver started at http://127.8.215.1:33161/ using document root <none> and password file <none>
I20260925 05:09:08.132211 20456 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:08.132428 20456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:08.132872 20456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:09:08 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:08.153046 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "f74b0312b687418883020eddd19302aa"
format_stamp: "Formatted at 2026-09-25 05:09:08 on dist-test-slave-wpdm"
server_key: "7dde31d9afe6fdff5936c82920c490af"
server_key_iv: "226d56921bc077620069f77e269e877f"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:08.154156 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "f74b0312b687418883020eddd19302aa"
format_stamp: "Formatted at 2026-09-25 05:09:08 on dist-test-slave-wpdm"
server_key: "7dde31d9afe6fdff5936c82920c490af"
server_key_iv: "226d56921bc077620069f77e269e877f"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:08.161305 20456 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 05:09:08.197494 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:08.199793 20456 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.006s sys 0.014s
I20260925 05:09:08.200099 20456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "f74b0312b687418883020eddd19302aa"
format_stamp: "Formatted at 2026-09-25 05:09:08 on dist-test-slave-wpdm"
server_key: "7dde31d9afe6fdff5936c82920c490af"
server_key_iv: "226d56921bc077620069f77e269e877f"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:08.200464 20456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:08.252136 20456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:08.257570 20456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:08.258070 20456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:08.261294 20456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:08.263284 20456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:08.263384 20456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:08.263567 20456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:08.263643 20456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:08.356132 20456 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:35929
I20260925 05:09:08.356184 20588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:35929 every 8 connection(s)
I20260925 05:09:08.357901 20456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:08.361236 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 20456
I20260925 05:09:08.361457 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:08.362255 9052 external_mini_cluster.cc:1428] Setting key 57f41bf385ccd7d5731ce2030aeeba85
Sep 25 05:09:08 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312947, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Sep 25 05:09:08 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:09:08.448717 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:08.361175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:55757 (local address 127.8.215.62:43887)
0925 05:09:08.377698 (+ 16523us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:08.377705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:08.382880 (+ 5175us) server_negotiation.cc:407] Connection header received
0925 05:09:08.387142 (+ 4262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:08.387149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:08.387257 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:08.387346 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:08.390814 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:08.392091 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:08.395973 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:08.396795 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:08.396933 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:08.440490 (+ 43557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:08.440500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:08.440508 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:08.440562 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:08.444517 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:08.445626 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:08.445632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:08.445635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:08.445800 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:08.446396 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:08.446401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:08.446404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:08.446771 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:08.446913 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:08.447947 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0925 05:09:08.448182 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16371,"thread_start_us":16533,"threads_started":1}
I20260925 05:09:08.451376 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:08.361913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43887 (local address 127.8.215.1:55757)
0925 05:09:08.382651 (+ 20738us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:08.382723 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:08.386696 (+ 3973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:08.388878 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:08.388889 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:08.389604 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:08.390536 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:08.390565 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:08.392341 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:08.392350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:08.395716 (+ 3366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:08.395738 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:08.405727 (+ 9989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:08.405750 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:08.406121 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:08.434883 (+ 28762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:08.434923 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:08.440132 (+ 5209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:08.444763 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:08.444776 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:08.444798 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:08.445464 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:08.445977 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:08.445983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:08.445987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:08.446258 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:08.447008 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:08.447018 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:08.447487 (+ 469us) client_negotiation.cc:769] Sending connection context
0925 05:09:08.449319 (+ 1832us) client_negotiation.cc:240] Negotiation successful
0925 05:09:08.449902 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20403,"thread_start_us":20153,"threads_started":1}
I20260925 05:09:08.453709 20589 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43887
I20260925 05:09:08.454058 20589 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:08.455116 20589 heartbeater.cc:511] Master 127.8.215.62:43887 requested a full tablet report, sending...
I20260925 05:09:08.457811 20376 ts_manager.cc:194] Registered new tserver with Master: f74b0312b687418883020eddd19302aa (127.8.215.1:35929)
I20260925 05:09:08.460053 20376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:55757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
W20260925 05:09:08.487926 9052 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.215.2: real 0.123s user 0.002s sys 0.019s
I20260925 05:09:08.488935 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:41527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43887
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:09:08 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312947, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
W20260925 05:09:09.061679 20595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:09.062238 20595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:09.062534 20595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:09.074792 20595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:09.075042 20595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:09.075325 20595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:09:09.091959 20595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.215.20:41527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:09:09.094151 20595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:09.096025 20595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:09.129881 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:09.129853 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:09.139213 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:09.141172 20595 server_base.cc:1034] running on GCE node
I20260925 05:09:09.142150 20595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:09.143509 20595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:09.144860 20595 hybrid_clock.cc:648] HybridClock initialized: now 1790312949144780 us; error 66 us; skew 500 ppm
I20260925 05:09:09.145350 20595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:09:09 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:09.150208 20595 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:09:09.152518 20595 webserver.cc:460] Webserver started at http://127.8.215.2:42683/ using document root <none> and password file <none>
I20260925 05:09:09.153465 20595 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:09.153607 20595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:09.153956 20595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:09:09 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:09.177431 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "58e079e6aa324342b10aaa1c5f677fc0"
format_stamp: "Formatted at 2026-09-25 05:09:09 on dist-test-slave-wpdm"
server_key: "ea79ed36946ba51873ab07bd3a2b3da9"
server_key_iv: "02014013c40cbd796e9067b55a34e4d3"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:09.178639 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "58e079e6aa324342b10aaa1c5f677fc0"
format_stamp: "Formatted at 2026-09-25 05:09:09 on dist-test-slave-wpdm"
server_key: "ea79ed36946ba51873ab07bd3a2b3da9"
server_key_iv: "02014013c40cbd796e9067b55a34e4d3"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:09.187029 20595 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.002s
I20260925 05:09:09.214737 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:09.216979 20595 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260925 05:09:09.217283 20595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "58e079e6aa324342b10aaa1c5f677fc0"
format_stamp: "Formatted at 2026-09-25 05:09:09 on dist-test-slave-wpdm"
server_key: "ea79ed36946ba51873ab07bd3a2b3da9"
server_key_iv: "02014013c40cbd796e9067b55a34e4d3"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:09.217651 20595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:09.276710 20595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:09.282552 20595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:09.282954 20595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:09.290679 20595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:09.299144 20595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:09.299342 20595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:09.299507 20595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:09.299598 20595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:09.380808 20595 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:46827
I20260925 05:09:09.380891 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:46827 every 8 connection(s)
I20260925 05:09:09.382864 20595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:09.385303 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 20595
I20260925 05:09:09.385483 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:09.386020 9052 external_mini_cluster.cc:1428] Setting key c053c71cbe418f3259812d9710011783
Sep 25 05:09:09 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
I20260925 05:09:09.440629 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.390860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:59131 (local address 127.8.215.62:43887)
0925 05:09:09.409744 (+ 18884us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:09.409750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:09.409779 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:09:09.414572 (+ 4793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:09.414582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:09.414702 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:09.414838 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:09.417344 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.418553 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.421400 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.422120 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.422234 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:09.427045 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:09.427052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:09.427057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:09.427104 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:09.430204 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:09.437730 (+ 7526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:09.437737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:09.437746 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:09.437896 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:09.438539 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:09.438550 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:09.438552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:09.438839 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:09.439006 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:09.439974 (+ 968us) server_negotiation.cc:299] Negotiation successful
0925 05:09:09.440229 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18747,"thread_start_us":19309,"threads_started":1}
I20260925 05:09:09.441965 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.388584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43887 (local address 127.8.215.2:59131)
0925 05:09:09.405675 (+ 17091us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:09.405773 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:09.409627 (+ 3854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:09.415341 (+ 5714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:09.415358 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:09.416126 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:09.417111 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:09.417143 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.418782 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.418790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:09.421206 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:09.421224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.422636 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.422645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:09.423023 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:09.423882 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:09.423928 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:09.426751 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:09.430356 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:09.430365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:09.430385 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:09.431031 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:09.438031 (+ 7000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:09.438039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:09.438044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:09.438412 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:09.438981 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:09.438990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:09.439509 (+ 519us) client_negotiation.cc:769] Sending connection context
0925 05:09:09.439868 (+ 359us) client_negotiation.cc:240] Negotiation successful
0925 05:09:09.440458 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16590,"thread_start_us":16279,"threads_started":1}
I20260925 05:09:09.443681 20732 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43887
I20260925 05:09:09.443990 20732 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:09.444875 20732 heartbeater.cc:511] Master 127.8.215.62:43887 requested a full tablet report, sending...
I20260925 05:09:09.446642 20376 ts_manager.cc:194] Registered new tserver with Master: 58e079e6aa324342b10aaa1c5f677fc0 (127.8.215.2:46827)
I20260925 05:09:09.447875 20376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:59131
I20260925 05:09:09.464146 20589 heartbeater.cc:503] Master 127.8.215.62:43887 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:09:09.467784 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:41527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43887
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:09.812584 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:09.812940 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:09.813130 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:09.821893 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:09.822039 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:09.822297 20738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:09:09.833999 20738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.215.20:41527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:09:09.835510 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:09.837024 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:09.851752 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:09.851873 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:09.851847 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:09.852646 20738 server_base.cc:1034] running on GCE node
I20260925 05:09:09.853323 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:09.854341 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:09.855581 20738 hybrid_clock.cc:648] HybridClock initialized: now 1790312949855524 us; error 55 us; skew 500 ppm
I20260925 05:09:09.855926 20738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:09:09 dist-test-slave-wpdm krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:09.859154 20738 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:09:09.860903 20738 webserver.cc:460] Webserver started at http://127.8.215.3:34349/ using document root <none> and password file <none>
I20260925 05:09:09.861622 20738 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:09.861754 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:09.862090 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:09:09 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:09.878672 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "47b1a8d5c7984fbdb22816c1afc53380"
format_stamp: "Formatted at 2026-09-25 05:09:09 on dist-test-slave-wpdm"
server_key: "01d50b13a10d428e0f0bceaf612e59c8"
server_key_iv: "b98acb6e1dcec360d77834fa51c30734"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:09.879681 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "47b1a8d5c7984fbdb22816c1afc53380"
format_stamp: "Formatted at 2026-09-25 05:09:09 on dist-test-slave-wpdm"
server_key: "01d50b13a10d428e0f0bceaf612e59c8"
server_key_iv: "b98acb6e1dcec360d77834fa51c30734"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:09.885401 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:09:09.902127 20753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:09.903743 20738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:09:09.903935 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "47b1a8d5c7984fbdb22816c1afc53380"
format_stamp: "Formatted at 2026-09-25 05:09:09 on dist-test-slave-wpdm"
server_key: "01d50b13a10d428e0f0bceaf612e59c8"
server_key_iv: "b98acb6e1dcec360d77834fa51c30734"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:09.904166 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:09.930035 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:09.934259 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:09.934623 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:09.935623 20738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:09.937076 20738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:09.937151 20738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:09.937234 20738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:09.937276 20738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:09.973043 20738 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:39877
I20260925 05:09:09.973049 20865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:39877 every 8 connection(s)
I20260925 05:09:09.974566 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:09:09.983644 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 20738
I20260925 05:09:09.983822 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:09.984246 9052 external_mini_cluster.cc:1428] Setting key 2bff21398b2768a42521e4854b0473e2
Sep 25 05:09:09 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:09:09.996168 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.977427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:48173 (local address 127.8.215.62:43887)
0925 05:09:09.978055 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:09.978061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:09.979011 (+ 950us) server_negotiation.cc:407] Connection header received
0925 05:09:09.981008 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:09.981013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:09.981122 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:09.981252 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:09.982734 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.983461 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.985479 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.986005 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.986162 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:09.989897 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:09.989903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:09.989907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:09.989943 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:09.992984 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:09.993852 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:09.993857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:09.993859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:09.993989 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:09.994520 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:09.994524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:09.994526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:09.994913 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:09.995034 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:09.995706 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 05:09:09.995912 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":368,"threads_started":1}
I20260925 05:09:09.997255 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.978090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43887 (local address 127.8.215.3:48173)
0925 05:09:09.978824 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:09.978873 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:09.980661 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:09.981502 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:09.981511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:09.981951 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:09.982543 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:09.982560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.983645 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.983650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:09.985325 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:09.985337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:09.986312 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:09.986320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:09.986603 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:09.987392 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:09.987433 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:09.989649 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:09.993134 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:09.993142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:09.993157 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:09.993657 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:09.994091 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:09.994094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:09.994096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:09.994308 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:09.995030 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:09.995036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:09.995382 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 05:09:09.995620 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 05:09:09.996024 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":293,"threads_started":1}
I20260925 05:09:09.998939 20866 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43887
I20260925 05:09:09.999317 20866 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:10.000118 20866 heartbeater.cc:511] Master 127.8.215.62:43887 requested a full tablet report, sending...
I20260925 05:09:10.001648 20376 ts_manager.cc:194] Registered new tserver with Master: 47b1a8d5c7984fbdb22816c1afc53380 (127.8.215.3:39877)
I20260925 05:09:10.002540 20376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:48173
I20260925 05:09:10.012357 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:10.030745 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.017365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.8.215.62:43887)
0925 05:09:10.017575 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.017584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.018113 (+ 529us) server_negotiation.cc:407] Connection header received
0925 05:09:10.018291 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.018295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.018371 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.018455 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:10.019960 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.020674 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.022086 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.022478 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.022577 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.025299 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:10.025304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:10.025307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:10.025340 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:10.027968 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.028670 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.028674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.028677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.028895 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.029263 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.029267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.029269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.029567 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:10.029832 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.030155 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.030343 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:09:10.035056 20376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:10.042508 20376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:10.079406 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.056082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35929 (local address 127.0.0.1:32930)
0925 05:09:10.068686 (+ 12604us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.068706 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:10.068874 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:10.070421 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:10.070434 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:10.070476 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:10.071036 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.071051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.073861 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.073868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.076389 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.076401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.078328 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.078334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.078515 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.078562 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:09:10.078715 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 05:09:10.078917 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12477,"mutex_wait_us":3,"thread_start_us":13997,"threads_started":1}
I20260925 05:09:10.080334 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.056338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.8.215.1:35929)
0925 05:09:10.065393 (+ 9055us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.065407 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.069046 (+ 3639us) server_negotiation.cc:407] Connection header received
0925 05:09:10.069196 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.069208 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.069564 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.069825 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:10.071307 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.073246 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.076543 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.078109 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.078243 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.078453 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.079827 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.080023 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8812,"thread_start_us":8661,"threads_started":1}
I20260925 05:09:10.081633 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.055605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:46827 (local address 127.0.0.1:57056)
0925 05:09:10.073225 (+ 17620us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.073250 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:10.073394 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:10.074432 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:10.074439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:10.074464 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:10.075051 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.075069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.076930 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.076935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.079300 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.079314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.080837 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.080843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.081037 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.081072 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:09:10.081146 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:09:10.081285 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17444,"mutex_wait_us":923,"thread_start_us":9096,"threads_started":1}
I20260925 05:09:10.082231 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.055977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.8.215.2:46827)
0925 05:09:10.073759 (+ 17782us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.073768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.073798 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:09:10.073911 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.073918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.074210 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.074447 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:10.075282 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.076790 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.079499 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.080573 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.081453 (+ 880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.081637 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.081717 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.081879 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17604,"thread_start_us":17616,"threads_started":1}
I20260925 05:09:10.082231 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.055640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.8.215.3:39877)
0925 05:09:10.074206 (+ 18566us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.074212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.074238 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:09:10.074304 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.074309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.074803 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.075013 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:10.075681 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.077122 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.079368 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.080963 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.081086 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.081327 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.081728 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.081878 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18412,"thread_start_us":18253,"threads_started":1}
I20260925 05:09:10.082512 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.055596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39877 (local address 127.0.0.1:59016)
0925 05:09:10.066857 (+ 11261us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.066916 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:10.067145 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:10.075030 (+ 7885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:10.075035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:10.075060 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:10.075476 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.075488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.077271 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.077278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.079160 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.079173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.081165 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.081171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.081336 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.081384 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:09:10.081468 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:09:10.081597 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10198,"mutex_wait_us":7,"thread_start_us":17873,"threads_started":1}
I20260925 05:09:10.085068 20524 tablet_service.cc:1467] Processing CreateTablet for tablet 96afe2e050d54e909985c2328e45d2ad (DEFAULT_TABLE table=test-table [id=f7761f181f62440bbfcaccc50669154e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:10.085827 20801 tablet_service.cc:1467] Processing CreateTablet for tablet 96afe2e050d54e909985c2328e45d2ad (DEFAULT_TABLE table=test-table [id=f7761f181f62440bbfcaccc50669154e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:10.086104 20667 tablet_service.cc:1467] Processing CreateTablet for tablet 96afe2e050d54e909985c2328e45d2ad (DEFAULT_TABLE table=test-table [id=f7761f181f62440bbfcaccc50669154e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:10.087321 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96afe2e050d54e909985c2328e45d2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:10.087327 20801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96afe2e050d54e909985c2328e45d2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:10.087437 20667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96afe2e050d54e909985c2328e45d2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:10.099203 20885 tablet_bootstrap.cc:492] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Bootstrap starting.
I20260925 05:09:10.102402 20886 tablet_bootstrap.cc:492] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Bootstrap starting.
I20260925 05:09:10.103098 20885 tablet_bootstrap.cc:654] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:10.103291 20887 tablet_bootstrap.cc:492] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Bootstrap starting.
I20260925 05:09:10.104619 20885 log.cc:824] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:10.106992 20886 tablet_bootstrap.cc:654] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:10.107123 20887 tablet_bootstrap.cc:654] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:10.108242 20886 log.cc:824] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:10.108493 20887 log.cc:824] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:10.108810 20885 tablet_bootstrap.cc:492] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: No bootstrap required, opened a new log
I20260925 05:09:10.109145 20885 ts_tablet_manager.cc:1397] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 05:09:10.111290 20887 tablet_bootstrap.cc:492] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: No bootstrap required, opened a new log
I20260925 05:09:10.111577 20887 ts_tablet_manager.cc:1397] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 05:09:10.111640 20886 tablet_bootstrap.cc:492] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: No bootstrap required, opened a new log
I20260925 05:09:10.111899 20886 ts_tablet_manager.cc:1397] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260925 05:09:10.119340 20887 raft_consensus.cc:348] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.119872 20887 raft_consensus.cc:374] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:10.119992 20887 raft_consensus.cc:729] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74b0312b687418883020eddd19302aa, State: Initialized, Role: FOLLOWER
I20260925 05:09:10.120061 20885 raft_consensus.cc:348] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.120571 20885 raft_consensus.cc:374] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:10.120709 20885 raft_consensus.cc:729] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b1a8d5c7984fbdb22816c1afc53380, State: Initialized, Role: FOLLOWER
I20260925 05:09:10.120699 20887 consensus_queue.cc:260] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.121333 20885 consensus_queue.cc:260] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.122785 20887 ts_tablet_manager.cc:1428] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260925 05:09:10.122788 20886 raft_consensus.cc:348] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.123394 20886 raft_consensus.cc:374] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:10.123531 20886 raft_consensus.cc:729] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e079e6aa324342b10aaa1c5f677fc0, State: Initialized, Role: FOLLOWER
I20260925 05:09:10.123847 20866 heartbeater.cc:503] Master 127.8.215.62:43887 was elected leader, sending a full tablet report...
I20260925 05:09:10.124258 20886 consensus_queue.cc:260] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.124575 20885 ts_tablet_manager.cc:1428] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260925 05:09:10.125507 20732 heartbeater.cc:503] Master 127.8.215.62:43887 was elected leader, sending a full tablet report...
I20260925 05:09:10.127235 20886 ts_tablet_manager.cc:1428] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260925 05:09:10.129902 20590 tablet.cc:2369] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:09:10.157720 20733 tablet.cc:2369] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:09:10.226325 20867 tablet.cc:2369] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:10.329643 20891 raft_consensus.cc:482] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:10.330060 20891 raft_consensus.cc:504] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.332039 20891 leader_election.cc:288] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47b1a8d5c7984fbdb22816c1afc53380 (127.8.215.3:39877), 58e079e6aa324342b10aaa1c5f677fc0 (127.8.215.2:46827)
I20260925 05:09:10.339625 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.332631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:53985 (local address 127.8.215.2:46827)
0925 05:09:10.332932 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.332942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.333775 (+ 833us) server_negotiation.cc:407] Connection header received
0925 05:09:10.333966 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.333970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.334031 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.334159 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:10.334822 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.335634 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.337687 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.338601 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.338749 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.338873 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.339189 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.339333 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 05:09:10.339661 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.332484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:35083 (local address 127.8.215.3:39877)
0925 05:09:10.332698 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.332706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.333683 (+ 977us) server_negotiation.cc:407] Connection header received
0925 05:09:10.333929 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.333934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.334001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.334068 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:10.334926 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.335612 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.337863 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.338617 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.338775 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.338891 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.339273 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.339411 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:09:10.339804 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.332543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:46827 (local address 127.8.215.1:53985)
0925 05:09:10.333619 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.333660 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:10.333794 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:10.334184 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:10.334189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:10.334215 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:10.334645 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.334655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.335792 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.335796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.337519 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.337531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.338847 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.338854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.339008 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.339035 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:09:10.339112 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:09:10.339246 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":4,"thread_start_us":772,"threads_started":1}
I20260925 05:09:10.339851 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.332475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:39877 (local address 127.8.215.1:35083)
0925 05:09:10.333522 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.333552 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:10.333760 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:10.334123 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:10.334133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:10.334161 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:10.334774 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.334789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.335757 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.335763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.337695 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:10.337710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.338899 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.338909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:10.339064 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.339114 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:09:10.339220 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 05:09:10.339361 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":74,"thread_start_us":456,"threads_started":1}
I20260925 05:09:10.341071 20821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96afe2e050d54e909985c2328e45d2ad" candidate_uuid: "f74b0312b687418883020eddd19302aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b1a8d5c7984fbdb22816c1afc53380" is_pre_election: true
I20260925 05:09:10.341043 20687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96afe2e050d54e909985c2328e45d2ad" candidate_uuid: "f74b0312b687418883020eddd19302aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e079e6aa324342b10aaa1c5f677fc0" is_pre_election: true
I20260925 05:09:10.341563 20821 raft_consensus.cc:2454] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74b0312b687418883020eddd19302aa in term 0.
I20260925 05:09:10.341563 20687 raft_consensus.cc:2454] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74b0312b687418883020eddd19302aa in term 0.
I20260925 05:09:10.342265 20478 leader_election.cc:302] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b1a8d5c7984fbdb22816c1afc53380, f74b0312b687418883020eddd19302aa; no voters:
I20260925 05:09:10.342718 20891 raft_consensus.cc:2781] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:10.342927 20891 raft_consensus.cc:482] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:10.343043 20891 raft_consensus.cc:3037] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:10.345258 20891 raft_consensus.cc:504] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.345937 20891 leader_election.cc:288] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [CANDIDATE]: Term 1 election: Requested vote from peers 47b1a8d5c7984fbdb22816c1afc53380 (127.8.215.3:39877), 58e079e6aa324342b10aaa1c5f677fc0 (127.8.215.2:46827)
I20260925 05:09:10.346459 20687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96afe2e050d54e909985c2328e45d2ad" candidate_uuid: "f74b0312b687418883020eddd19302aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e079e6aa324342b10aaa1c5f677fc0"
I20260925 05:09:10.346460 20821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96afe2e050d54e909985c2328e45d2ad" candidate_uuid: "f74b0312b687418883020eddd19302aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b1a8d5c7984fbdb22816c1afc53380"
I20260925 05:09:10.346675 20687 raft_consensus.cc:3037] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:10.346668 20821 raft_consensus.cc:3037] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:10.348979 20687 raft_consensus.cc:2454] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74b0312b687418883020eddd19302aa in term 1.
I20260925 05:09:10.349437 20479 leader_election.cc:302] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e079e6aa324342b10aaa1c5f677fc0, f74b0312b687418883020eddd19302aa; no voters:
I20260925 05:09:10.349834 20891 raft_consensus.cc:2781] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:10.349821 20821 raft_consensus.cc:2454] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74b0312b687418883020eddd19302aa in term 1.
I20260925 05:09:10.350664 20891 raft_consensus.cc:686] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 1 LEADER]: Becoming Leader. State: Replica: f74b0312b687418883020eddd19302aa, State: Running, Role: LEADER
I20260925 05:09:10.351225 20891 consensus_queue.cc:237] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } }
I20260925 05:09:10.355968 20375 catalog_manager.cc:5495] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa reported cstate change: term changed from 0 to 1, leader changed from <none> to f74b0312b687418883020eddd19302aa (127.8.215.1). New cstate: current_term: 1 leader_uuid: "f74b0312b687418883020eddd19302aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74b0312b687418883020eddd19302aa" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35929 } health_report { overall_health: HEALTHY } } }
I20260925 05:09:10.434983 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.429531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.8.215.1:35929)
0925 05:09:10.429748 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.429753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.429768 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:09:10.429861 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.429864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.429950 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.430107 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:10.430484 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.431183 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.432659 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.433159 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.433304 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.433866 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:10.434398 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:10.434571 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.434626 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.434707 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:09:10.443584 20687 raft_consensus.cc:1261] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 1 FOLLOWER]: Refusing update from remote peer f74b0312b687418883020eddd19302aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:10.443579 20821 raft_consensus.cc:1261] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 1 FOLLOWER]: Refusing update from remote peer f74b0312b687418883020eddd19302aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:10.444592 20896 consensus_queue.cc:1035] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b1a8d5c7984fbdb22816c1afc53380" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:10.445026 20891 consensus_queue.cc:1035] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e079e6aa324342b10aaa1c5f677fc0" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:10.458832 20901 mvcc.cc:205] Tried to move back new op lower bound from 7333121845006569472 to 7333121844642971648. Current Snapshot: MvccSnapshot[applied={T|T < 7333121845006569472}]
I20260925 05:09:10.474382 20375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:10.474677 20375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:10.478715 20375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e438e1ba03f4fc8867d611ea6e28629: Sending DeleteTablet for 3 replicas of tablet 96afe2e050d54e909985c2328e45d2ad
I20260925 05:09:10.479991 20801 tablet_service.cc:1514] Processing DeleteTablet for tablet 96afe2e050d54e909985c2328e45d2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:10 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:59016
I20260925 05:09:10.480006 20667 tablet_service.cc:1514] Processing DeleteTablet for tablet 96afe2e050d54e909985c2328e45d2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:10 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:57056
I20260925 05:09:10.480094 20524 tablet_service.cc:1514] Processing DeleteTablet for tablet 96afe2e050d54e909985c2328e45d2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:10 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:32930
I20260925 05:09:10.483142 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 20456
I20260925 05:09:10.492690 20911 tablet_replica.cc:330] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: stopping tablet replica
I20260925 05:09:10.493253 20909 tablet_replica.cc:330] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: stopping tablet replica
I20260925 05:09:10.493264 20911 raft_consensus.cc:2229] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:10.493672 20909 raft_consensus.cc:2229] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:09:10.493782 20911 raft_consensus.cc:2258] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:10.494700 20909 raft_consensus.cc:2258] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:10.496485 20911 ts_tablet_manager.cc:1905] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:10.497418 20909 ts_tablet_manager.cc:1905] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:10.498157 20910 tablet_replica.cc:330] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: stopping tablet replica
I20260925 05:09:10.498728 20910 raft_consensus.cc:2229] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:10.499934 20910 raft_consensus.cc:2258] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:10.504406 20911 ts_tablet_manager.cc:1918] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:10.504662 20911 log.cc:1196] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/96afe2e050d54e909985c2328e45d2ad
I20260925 05:09:10.505278 20911 ts_tablet_manager.cc:1939] T 96afe2e050d54e909985c2328e45d2ad P 47b1a8d5c7984fbdb22816c1afc53380: Deleting consensus metadata
I20260925 05:09:10.507539 20361 catalog_manager.cc:4841] TS 47b1a8d5c7984fbdb22816c1afc53380 (127.8.215.3:39877): tablet 96afe2e050d54e909985c2328e45d2ad (table test-table [id=f7761f181f62440bbfcaccc50669154e]) successfully deleted
I20260925 05:09:10.507588 20910 ts_tablet_manager.cc:1905] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:10.509814 20909 ts_tablet_manager.cc:1918] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:10.510030 20909 log.cc:1196] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/96afe2e050d54e909985c2328e45d2ad
I20260925 05:09:10.510655 20909 ts_tablet_manager.cc:1939] T 96afe2e050d54e909985c2328e45d2ad P f74b0312b687418883020eddd19302aa: Deleting consensus metadata
I20260925 05:09:10.512789 20360 catalog_manager.cc:4841] TS f74b0312b687418883020eddd19302aa (127.8.215.1:35929): tablet 96afe2e050d54e909985c2328e45d2ad (table test-table [id=f7761f181f62440bbfcaccc50669154e]) successfully deleted
I20260925 05:09:10.514719 20910 ts_tablet_manager.cc:1918] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:10.515327 20910 log.cc:1196] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/96afe2e050d54e909985c2328e45d2ad
I20260925 05:09:10.515872 20910 ts_tablet_manager.cc:1939] T 96afe2e050d54e909985c2328e45d2ad P 58e079e6aa324342b10aaa1c5f677fc0: Deleting consensus metadata
Sep 25 05:09:10 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:09:10.517702 20362 catalog_manager.cc:4841] TS 58e079e6aa324342b10aaa1c5f677fc0 (127.8.215.2:46827): tablet 96afe2e050d54e909985c2328e45d2ad (table test-table [id=f7761f181f62440bbfcaccc50669154e]) successfully deleted
I20260925 05:09:10.525259 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.485219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.8.215.1:35929)
0925 05:09:10.485447 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.485456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.499115 (+ 13659us) server_negotiation.cc:407] Connection header received
0925 05:09:10.499347 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.499353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.499431 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.499541 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:10.502638 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.504049 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.506292 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.506945 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.507095 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.518444 (+ 11349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:10.518459 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:10.518474 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:10.518514 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:10.521802 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.522793 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.522800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.522803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.522957 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.523449 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.523452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.523454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.523881 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:10.524126 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.524554 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.524846 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:09:10.525961 20584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:09:10.627008 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 20456
I20260925 05:09:10.646117 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 20595
Sep 25 05:09:10 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:09:10.661497 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.647032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.8.215.2:46827)
0925 05:09:10.647236 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.647243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.647265 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:09:10.647334 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.647340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.647420 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.647529 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:10.648809 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.649610 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.650854 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.651257 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.651353 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.655705 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:10.655721 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:10.655737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:10.655771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:10.658867 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.659490 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.659497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.659499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.659620 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.659971 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.659976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.659979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.660399 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:10.660606 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.660952 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.661105 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:09:10.662187 20727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:09:10.752686 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 20595
I20260925 05:09:10.769652 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 20738
Sep 25 05:09:10 dist-test-slave-wpdm krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:09:10.786602 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.770402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.8.215.3:39877)
0925 05:09:10.770568 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:10.770572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:10.770591 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:09:10.770745 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:10.770749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:10.770818 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:10.770905 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:10.772325 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.773259 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.774689 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:10.775138 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:10.775260 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:10.780212 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:10.780229 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:10.780248 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:10.780295 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:10.783392 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.784181 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.784194 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.784197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.784333 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:10.784939 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:10.784943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:10.784946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:10.785387 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:10.785695 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:10.786030 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 05:09:10.786251 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:09:10.787235 20861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:09:10.877986 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 20738
I20260925 05:09:10.894907 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 20344
I20260925 05:09:10.896216 20406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:09:11.024219 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 20344
I20260925 05:09:11.051180 9052 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:09:11.387156 9052 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:09:11.387362 9052 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:09:11.733163 9052 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:09:11.733 UTC [19930] LOG: received smart shutdown request
2026-09-25 05:09:11.737 UTC [19930] LOG: background worker "logical replication launcher" (PID 19939) exited with exit code 1
2026-09-25 05:09:11.737 UTC [19934] LOG: shutting down
2026-09-25 05:09:11.921 UTC [19930] LOG: database system is shut down
2026-09-25T05:09:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49827 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:12 dist-test-slave-wpdm krb5kdc[20925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:12 dist-test-slave-wpdm krb5kdc[20925](info): set up 2 sockets
Sep 25 05:09:12 dist-test-slave-wpdm krb5kdc[20925](info): commencing operation
krb5kdc: starting...
W20260925 05:09:13.170707 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:09:13 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/rangeradmin_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/rangeradmin_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/rangerlookup_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/rangerlookup_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
I20260925 05:09:13.535837 9052 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjaqqHJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/postgres -l logfile start
2026-09-25 05:09:16.162 UTC [20955] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:09:16.162 UTC [20955] LOG: listening on IPv4 address "127.8.215.20", port 37447
2026-09-25 05:09:16.166 UTC [20955] LOG: listening on Unix socket "/tmp/.s.PGSQL.37447"
2026-09-25 05:09:16.195 UTC [20958] LOG: database system was shut down at 2026-09-25 05:09:14 UTC
2026-09-25 05:09:16.199 UTC [20955] LOG: database system is ready to accept connections
I20260925 05:09:17.177229 9052 mini_postgres.cc:96] Postgres bound to 37447
2026-09-25 05:09:17.196 UTC [20966] FATAL: database "slave" does not exist
127.8.215.20:37447 - accepting connections
I20260925 05:09:17.197372 9052 mini_ranger.cc:161] Starting Ranger...
I20260925 05:09:17.224565 9052 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:09:17.725896 9052 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:09:17.726022 9052 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-admin
2026-09-25 05:09:18,031 [I] DB FLAVOR :POSTGRES
2026-09-25 05:09:18,032 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:09:18,032 [I] Checking connection..
2026-09-25 05:09:18,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:09:18,286 [I] Checking connection passed.
2026-09-25 05:09:18,286 [I] --------- Verifying version history table ---------
2026-09-25 05:09:18,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:09:18,551 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:09:18,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:09:18,796 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:09:18,797 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:09:18,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:09:19.080 UTC [21030] WARNING: there is no transaction in progress
2026-09-25 05:09:19,086 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:09:19,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:09:19,352 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:09:19,353 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:09:19,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:09:19,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:09:19,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wpdm.c.gcp-upstream.internal','N') ;"
2026-09-25 05:09:20,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:09:20,343 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:09:20,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:09:20,577 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:09:20,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:09:20,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:09:21,075 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:09:21,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:09:22.047 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.063 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.080 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.096 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.113 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.131 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.187 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.194 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.209 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.219 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.230 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.245 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.254 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.262 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.268 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22.276 UTC [21138] WARNING: there is no transaction in progress
2026-09-25 05:09:22,953 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:09:22,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:09:23,214 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:09:23,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:09:23,470 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:09:23,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:09:23,707 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:09:23,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:09:23,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:09:24,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wpdm.c.gcp-upstream.internal';"
2026-09-25 05:09:24,464 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:09:24,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:09:24,723 [I] DB_PATCHES have already been applied
I20260925 05:09:24.729554 9052 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjaqqHJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:09:24.729643 9052 mini_ranger.cc:193] Using host: 127.8.215.20
I20260925 05:09:24.745746 9052 mini_ranger.cc:227] Ranger admin URL: http://127.8.215.20:40573
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:09:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:09:25.923790 9052 mini_ranger.cc:160] Time spent starting Ranger: real 8.726s user 0.003s sys 0.041s
Sep 25, 2026 5:09:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:09:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 5:09:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:09:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:09:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:09:48.780090 9052 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.215.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.215.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/rangerkms_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.215.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/rangerkms_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.215.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.215.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.215.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/HTTP_127.8.215.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:09:48 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:09:48.973248 9052 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:09:48.995560 9052 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:09:49.889731 9052 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:09:49.889854 9052 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms
I20260925 05:09:49.889905 9052 mini_ranger_kms.cc:227] Using postgres at 127.8.215.20:37447
2026-09-25 05:09:50,261 [I] DB FLAVOR :POSTGRES
2026-09-25 05:09:50,261 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:09:50,261 [I] Checking connection
2026-09-25 05:09:50,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:09:50,535 [I] connection success
2026-09-25 05:09:50,535 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:09:50,535 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:09:50,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:09:50,808 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:09:50,808 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:09:50,809 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:09:50,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.215.20:37447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:09:51,134 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:09:51.652230 9052 mini_ranger_kms.cc:310] Created kms service
I20260925 05:09:51.884096 9052 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:09:51.936615 9052 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:09:51 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:52.340425 9052 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:09:52.340788 9052 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjaqqHJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjaqqHJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:09:52.340852 9052 mini_ranger_kms.cc:240] Using host: 127.8.215.20
I20260925 05:09:52.360915 9052 mini_ranger_kms.cc:276] Ranger KMS PID: 21319
I20260925 05:09:52.361061 9052 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.215.20:37007
Sep 25, 2026 5:09:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:09:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:09:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:09:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:09:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:09:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 05:09:54 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312994, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.215.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:54.767154 9052 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.794s user 0.091s sys 0.297s
I20260925 05:09:54.767450 9052 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:09:54.767659 9052 mini_ranger_kms.cc:401] 127.8.215.20:37007/kms/v1/keys
Sep 25 05:09:55 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312994, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.215.20@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
Sep 25 05:09:55 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312994, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.215.20@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
Sep 25 05:09:57 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:09:57.684463 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:42281
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:42281
--ranger_config_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:58.020053 21369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:58.020356 21369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:58.020444 21369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:58.028158 21369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:58.028282 21369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:58.028328 21369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:58.028378 21369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:58.028424 21369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:58.039012 21369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.215.20:37007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:42281
--ranger_config_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:09:58.040270 21369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:58.041662 21369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:58.054186 21374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:58.054276 21375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:58.054312 21377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:58.054980 21369 server_base.cc:1034] running on GCE node
I20260925 05:09:58.055791 21369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:58.057327 21369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:58.058562 21369 hybrid_clock.cc:648] HybridClock initialized: now 1790312998058521 us; error 59 us; skew 500 ppm
I20260925 05:09:58.058940 21369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:09:58 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312998, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:58.062156 21369 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:09:58.063637 21369 webserver.cc:460] Webserver started at http://127.8.215.62:39371/ using document root <none> and password file <none>
I20260925 05:09:58.064270 21369 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:58.064363 21369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:58.064608 21369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:09:58 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312998, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:58.130453 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "7c7f43e69c4a48ce93a9da761a0a2dde"
format_stamp: "Formatted at 2026-09-25 05:09:58 on dist-test-slave-wpdm"
server_key: "bb52645dea59944ebb5e346e3aa921fe"
server_key_iv: "fe9bdcca41d64ccfb4681e985a2a833b"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:58.131408 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "7c7f43e69c4a48ce93a9da761a0a2dde"
format_stamp: "Formatted at 2026-09-25 05:09:58 on dist-test-slave-wpdm"
server_key: "bb52645dea59944ebb5e346e3aa921fe"
server_key_iv: "fe9bdcca41d64ccfb4681e985a2a833b"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:58.137259 21369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:09:58.158542 21384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:58.160265 21369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 05:09:58.160458 21369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "7c7f43e69c4a48ce93a9da761a0a2dde"
format_stamp: "Formatted at 2026-09-25 05:09:58 on dist-test-slave-wpdm"
server_key: "bb52645dea59944ebb5e346e3aa921fe"
server_key_iv: "fe9bdcca41d64ccfb4681e985a2a833b"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:58.160717 21369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:58.182752 21369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:58.187310 21369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:58.187728 21369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:58.208155 21369 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:42281
I20260925 05:09:58.208174 21435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:42281 every 8 connection(s)
I20260925 05:09:58.209754 21369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:58.211428 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 21369
I20260925 05:09:58.211694 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:09:58.212754 9052 external_mini_cluster.cc:1428] Setting key 91784e77c073be6491741e4410830bd4
I20260925 05:09:58.215005 21436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:58.229482 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Bootstrap starting.
I20260925 05:09:58.233518 21436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:58.234854 21436 log.cc:824] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:58.238231 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: No bootstrap required, opened a new log
Sep 25 05:09:58 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:09:58.249049 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:58.215167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.8.215.62:42281)
0925 05:09:58.225300 (+ 10133us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:58.225317 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:58.229154 (+ 3837us) server_negotiation.cc:407] Connection header received
0925 05:09:58.231156 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:58.231178 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:58.231588 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:58.231995 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:58.233509 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:58.234917 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:58.236680 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:58.237411 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:58.237623 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:58.241516 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:58.241529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:58.241549 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:58.241604 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:58.244698 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:58.245471 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:58.245478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:58.245484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:58.245717 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:58.246115 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:58.246126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:58.246129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:58.246704 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:58.247028 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:58.247519 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 05:09:58.247978 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9839,"thread_start_us":9566,"threads_started":1}
I20260925 05:09:58.249207 21436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:09:58.249631 21436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:58.249735 21436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7f43e69c4a48ce93a9da761a0a2dde, State: Initialized, Role: FOLLOWER
I20260925 05:09:58.251951 21436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:09:58.252341 21436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:58.252482 21436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:58.252660 21436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:58.255643 21436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:09:58.256227 21436 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7f43e69c4a48ce93a9da761a0a2dde; no voters:
I20260925 05:09:58.256915 21436 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:58.257056 21441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:58.260180 21441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 LEADER]: Becoming Leader. State: Replica: 7c7f43e69c4a48ce93a9da761a0a2dde, State: Running, Role: LEADER
I20260925 05:09:58.260949 21441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:09:58.262900 21436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:58.264046 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } } }
I20260925 05:09:58.264509 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:58.264043 21442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7f43e69c4a48ce93a9da761a0a2dde. Latest consensus state: current_term: 1 leader_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } } }
I20260925 05:09:58.265165 21442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:58.269729 21446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:58.274791 21446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:58.292739 21446 catalog_manager.cc:1347] Generated new cluster ID: fb965918f23844eaa43f36460fd066f4
I20260925 05:09:58.292877 21446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:58.329725 21446 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:58.331785 21446 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:58.347289 21436 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:09:58.352255 21446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Generated new TSK 0
I20260925 05:09:58.353452 21446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:09:59.129429 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42281
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:09:59 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 05:09:59.563673 21481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:59.564159 21481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:59.564375 21481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:59.576468 21481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:59.576700 21481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:59.576953 21481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:09:59.593122 21481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.215.20:37007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:09:59.595082 21481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:59.596885 21481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:59.613967 21491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:59.615701 21492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:59.624157 21481 server_base.cc:1034] running on GCE node
W20260925 05:09:59.624416 21494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:59.625373 21481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:59.626724 21481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:59.628054 21481 hybrid_clock.cc:648] HybridClock initialized: now 1790312999627964 us; error 72 us; skew 500 ppm
I20260925 05:09:59.628460 21481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:09:59 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:59.632817 21481 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:09:59.637301 21481 webserver.cc:460] Webserver started at http://127.8.215.1:33479/ using document root <none> and password file <none>
I20260925 05:09:59.638242 21481 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:59.638378 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:59.638722 21481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:09:59 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:09:59.661419 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "aad512203eb34ca0a93207adfcea9953"
format_stamp: "Formatted at 2026-09-25 05:09:59 on dist-test-slave-wpdm"
server_key: "831a888bac306a4405a3bc53929380e7"
server_key_iv: "3ec49c9ea5044f94447e08023a64cc95"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:59.662576 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "aad512203eb34ca0a93207adfcea9953"
format_stamp: "Formatted at 2026-09-25 05:09:59 on dist-test-slave-wpdm"
server_key: "831a888bac306a4405a3bc53929380e7"
server_key_iv: "3ec49c9ea5044f94447e08023a64cc95"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:59.674247 21481 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s
I20260925 05:09:59.700186 21501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:59.701939 21481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:09:59.702138 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "aad512203eb34ca0a93207adfcea9953"
format_stamp: "Formatted at 2026-09-25 05:09:59 on dist-test-slave-wpdm"
server_key: "831a888bac306a4405a3bc53929380e7"
server_key_iv: "3ec49c9ea5044f94447e08023a64cc95"
server_key_version: "kuduclusterkey@0"
I20260925 05:09:59.702379 21481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:59.742678 21481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:59.748533 21481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:59.749004 21481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:59.750396 21481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:59.752905 21481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:59.753072 21481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:59.753283 21481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:59.753417 21481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:59.822068 21481 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:34793
I20260925 05:09:59.822098 21613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:34793 every 8 connection(s)
I20260925 05:09:59.823920 21481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:59.834231 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 21481
I20260925 05:09:59.834530 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:59.835520 9052 external_mini_cluster.cc:1428] Setting key a930a2a1861a406e2f899679b8b9aacd
Sep 25 05:09:59 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:09:59.867916 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:59.831402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:47379 (local address 127.8.215.62:42281)
0925 05:09:59.837294 (+ 5892us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:59.837301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:59.842100 (+ 4799us) server_negotiation.cc:407] Connection header received
0925 05:09:59.842171 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:59.842176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:59.842264 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:59.842380 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:59.844472 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:59.845585 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:59.847898 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:59.848561 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:59.848663 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:59.855213 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:59.855224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:59.855234 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:59.855289 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:59.861397 (+ 6108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:59.863114 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:59.863124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:59.863130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:59.863317 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:59.864029 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:59.864037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:59.864044 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:59.864523 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:59.864683 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:59.866959 (+ 2276us) server_negotiation.cc:299] Negotiation successful
0925 05:09:59.867354 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5683,"thread_start_us":5507,"threads_started":1}
I20260925 05:09:59.870944 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:59.832394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42281 (local address 127.8.215.1:47379)
0925 05:09:59.839501 (+ 7107us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:59.839551 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:59.841717 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:59.842850 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:59.842863 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:59.843461 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:59.844263 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:59.844296 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:59.845785 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:59.845789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:59.847705 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:59.847723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:59.849049 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:59.849055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:59.849351 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:59.850205 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:59.850235 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:59.854830 (+ 4595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:59.861940 (+ 7110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:59.861955 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:59.861981 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:59.862918 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:59.863502 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:59.863510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:59.863518 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:59.863886 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:59.864843 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:59.864855 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:59.866395 (+ 1540us) client_negotiation.cc:769] Sending connection context
0925 05:09:59.868559 (+ 2164us) client_negotiation.cc:240] Negotiation successful
0925 05:09:59.869280 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5164,"thread_start_us":5726,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
I20260925 05:09:59.873363 21614 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42281
I20260925 05:09:59.873708 21614 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
I20260925 05:09:59.874647 21614 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:09:59.877786 21401 ts_manager.cc:194] Registered new tserver with Master: aad512203eb34ca0a93207adfcea9953 (127.8.215.1:34793)
I20260925 05:09:59.880412 21401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:47379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:09:59.919340 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42281
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:09:59 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
Sep 25 05:10:00 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
W20260925 05:10:00.401268 21620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:00.401866 21620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:00.402103 21620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:00.417346 21620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:00.417630 21620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:00.417907 21620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:10:00.435489 21620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.215.20:37007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:00.437949 21620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:00.439980 21620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:00.475270 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:00.475265 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:00.481596 21620 server_base.cc:1034] running on GCE node
W20260925 05:10:00.484549 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:00.485525 21620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:00.488956 21620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:00.492435 21620 hybrid_clock.cc:648] HybridClock initialized: now 1790313000492356 us; error 67 us; skew 500 ppm
I20260925 05:10:00.492974 21620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:00 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313000, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:00.497466 21620 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:10:00.513276 21620 webserver.cc:460] Webserver started at http://127.8.215.2:39321/ using document root <none> and password file <none>
I20260925 05:10:00.514215 21620 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:00.514423 21620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:00.514816 21620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:10:00 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313000, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:00.537627 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "7d724666de584da08f17dd1191b22ac6"
format_stamp: "Formatted at 2026-09-25 05:10:00 on dist-test-slave-wpdm"
server_key: "be515d9c500409b0354b64de79fae3db"
server_key_iv: "e901c886b0d2c3f15f16aecac01d45a5"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:00.538568 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "7d724666de584da08f17dd1191b22ac6"
format_stamp: "Formatted at 2026-09-25 05:10:00 on dist-test-slave-wpdm"
server_key: "be515d9c500409b0354b64de79fae3db"
server_key_iv: "e901c886b0d2c3f15f16aecac01d45a5"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:00.545166 21620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 05:10:00.579926 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:00.582108 21620 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260925 05:10:00.582324 21620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "7d724666de584da08f17dd1191b22ac6"
format_stamp: "Formatted at 2026-09-25 05:10:00 on dist-test-slave-wpdm"
server_key: "be515d9c500409b0354b64de79fae3db"
server_key_iv: "e901c886b0d2c3f15f16aecac01d45a5"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:00.582605 21620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:00.610512 21620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:00.615255 21620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:00.615621 21620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:00.616667 21620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:00.618361 21620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:00.618443 21620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:00.618580 21620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:00.618636 21620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:00.657727 21620 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:46445
I20260925 05:10:00.657765 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:46445 every 8 connection(s)
I20260925 05:10:00.659124 21620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:00.665159 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 21620
I20260925 05:10:00.665329 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:00.665859 9052 external_mini_cluster.cc:1428] Setting key 947b77b67a2e239a1f614ef453d0c9f1
Sep 25 05:10:00 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313000, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:00.681847 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:00.662047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50711 (local address 127.8.215.62:42281)
0925 05:10:00.662581 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:00.662587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:00.663561 (+ 974us) server_negotiation.cc:407] Connection header received
0925 05:10:00.665989 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:00.665996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:00.666100 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:00.666230 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:00.667934 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:00.669029 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:00.670708 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:00.671463 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:00.671619 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:00.675492 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:00.675501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:00.675508 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:00.675555 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:00.678634 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:00.679448 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:00.679456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:00.679459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:00.679596 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:00.680028 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:00.680033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:00.680037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:00.680318 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:00.680483 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:00.681222 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 05:10:00.681457 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":243,"threads_started":1}
I20260925 05:10:00.682859 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:00.662556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42281 (local address 127.8.215.2:50711)
0925 05:10:00.663329 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:00.663369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:00.665708 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:00.666439 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:00.666473 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:00.667107 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:00.667727 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:00.667746 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:00.669190 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:00.669194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:00.670502 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:00.670515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:00.671719 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:00.671725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:00.671937 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:00.672702 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:00.672875 (+ 173us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:00.675255 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:00.678811 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:00.678820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:00.678845 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:00.679327 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:00.679708 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:00.679713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:00.679716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:00.679914 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:00.680448 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:00.680457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:00.680863 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 05:10:00.681134 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 05:10:00.681699 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":282,"threads_started":1}
I20260925 05:10:00.684474 21757 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42281
I20260925 05:10:00.684885 21757 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:00.685840 21757 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:10:00.687566 21401 ts_manager.cc:194] Registered new tserver with Master: 7d724666de584da08f17dd1191b22ac6 (127.8.215.2:46445)
I20260925 05:10:00.688491 21401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:50711
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:10:00.723874 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42281
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:00.884565 21614 heartbeater.cc:503] Master 127.8.215.62:42281 was elected leader, sending a full tablet report...
W20260925 05:10:01.067646 21763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:01.067974 21763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:01.068147 21763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:01.076712 21763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:01.076956 21763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:01.077150 21763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:10:01.088541 21763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.215.20:37007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:42281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:01.090046 21763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:01.091602 21763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:01.106230 21771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:01.106354 21768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:01.106382 21769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:01.106992 21763 server_base.cc:1034] running on GCE node
I20260925 05:10:01.107566 21763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:01.108525 21763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:01.109760 21763 hybrid_clock.cc:648] HybridClock initialized: now 1790313001109712 us; error 42 us; skew 500 ppm
I20260925 05:10:01.110119 21763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:01 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313001, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:01.117878 21763 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:10:01.119437 21763 webserver.cc:460] Webserver started at http://127.8.215.3:44719/ using document root <none> and password file <none>
I20260925 05:10:01.120167 21763 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:01.120301 21763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:01.120623 21763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:10:01 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313001, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:01.136801 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a"
format_stamp: "Formatted at 2026-09-25 05:10:01 on dist-test-slave-wpdm"
server_key: "33707e74023533a21a358608ecc4fbcf"
server_key_iv: "91c182d0602101672900414eab9aff01"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:01.137703 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a"
format_stamp: "Formatted at 2026-09-25 05:10:01 on dist-test-slave-wpdm"
server_key: "33707e74023533a21a358608ecc4fbcf"
server_key_iv: "91c182d0602101672900414eab9aff01"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:01.143051 21763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:10:01.158821 21778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:01.160779 21763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 05:10:01.160982 21763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a"
format_stamp: "Formatted at 2026-09-25 05:10:01 on dist-test-slave-wpdm"
server_key: "33707e74023533a21a358608ecc4fbcf"
server_key_iv: "91c182d0602101672900414eab9aff01"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:01.161229 21763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:01.194069 21763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:01.198330 21763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:01.198659 21763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:01.199628 21763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:01.201113 21763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:01.201218 21763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:01.201367 21763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:01.201442 21763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:01.234711 21763 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:37921
I20260925 05:10:01.234762 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:37921 every 8 connection(s)
I20260925 05:10:01.236042 21763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:01.238251 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 21763
I20260925 05:10:01.238497 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:01.239112 9052 external_mini_cluster.cc:1428] Setting key 195a545e281f1988301fac22c6eed1e5
Sep 25 05:10:01 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313001, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:01.257884 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:01.239127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:39191 (local address 127.8.215.62:42281)
0925 05:10:01.239776 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:01.239782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:01.240794 (+ 1012us) server_negotiation.cc:407] Connection header received
0925 05:10:01.243233 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:01.243237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:01.243315 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:01.243405 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:01.244920 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.245630 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.247188 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.247621 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.247714 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:01.251666 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:01.251671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:01.251674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:01.251706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:01.254647 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.255483 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.255489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.255493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.255631 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.256096 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.256105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.256107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.256364 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:01.256544 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:01.257293 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 05:10:01.257520 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":269,"threads_started":1}
I20260925 05:10:01.258812 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:01.239650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42281 (local address 127.8.215.3:39191)
0925 05:10:01.240618 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:01.240676 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:01.242973 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:01.243660 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:01.243669 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:01.244095 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:01.244701 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:01.244720 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.245769 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.245774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:01.247054 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:01.247068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.247822 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.247827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:01.248041 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:01.248695 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:01.248720 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:01.251429 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:01.254792 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:01.254801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:01.254820 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:01.255356 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:01.255745 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:01.255751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:01.255754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:01.255970 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:01.256496 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:01.256505 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:01.256919 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 05:10:01.257182 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 05:10:01.257692 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":374,"threads_started":1}
I20260925 05:10:01.260381 21891 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42281
I20260925 05:10:01.260700 21891 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:01.261507 21891 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:10:01.263033 21401 ts_manager.cc:194] Registered new tserver with Master: b2d0ccaf7bf14d0dabeabc8852aaac0a (127.8.215.3:37921)
I20260925 05:10:01.263983 21401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:39191
I20260925 05:10:01.267458 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:01.269181 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 21481
Sep 25 05:10:01 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:10:01.285730 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:01.270069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.8.215.1:34793)
0925 05:10:01.270709 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:01.270717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:01.270745 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:10:01.270902 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:01.270909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:01.271103 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:01.271282 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:01.272709 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.273764 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.275191 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.275734 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.275878 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:01.280181 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:01.280192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:01.280206 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:01.280237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:01.283110 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.283833 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.283840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.283843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.283983 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.284315 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.284318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.284320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.284628 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:01.284933 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:01.285129 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 05:10:01.285333 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":284,"threads_started":1}
I20260925 05:10:01.287024 21609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:01.371732 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 21481
I20260925 05:10:01.387899 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 21620
Sep 25 05:10:01 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:10:01.404400 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:01.388933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.8.215.2:46445)
0925 05:10:01.389540 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:01.389548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:01.389578 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:10:01.389650 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:01.389656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:01.389887 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:01.390026 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:01.391444 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.392528 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.393874 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.394433 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.394533 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:01.398687 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:01.398700 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:01.398716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:01.398751 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:01.401920 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.402615 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.402622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.402625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.402756 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.403092 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.403095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.403097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.403441 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:01.403665 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:01.403926 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 05:10:01.404116 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":347,"threads_started":1}
I20260925 05:10:01.406095 21752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:01.498791 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 21620
I20260925 05:10:01.515041 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 21763
Sep 25 05:10:01 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:10:01.531639 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:01.516140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.8.215.3:37921)
0925 05:10:01.516774 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:01.516788 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:01.516832 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:10:01.516893 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:01.516899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:01.517122 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:01.517286 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:01.518538 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.519563 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.520821 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:01.521396 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:01.521495 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:01.525723 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:01.525735 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:01.525750 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:01.525786 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:01.528615 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.529463 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.529477 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.529479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.529610 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:01.530017 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:01.530026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:01.530028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:01.530376 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:01.530704 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:01.531016 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 05:10:01.531286 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":284,"threads_started":1}
I20260925 05:10:01.533006 21886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:01.616109 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 21763
I20260925 05:10:01.632067 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 21369
I20260925 05:10:01.633499 21431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:01.740307 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 21369
I20260925 05:10:01.762172 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:42281
--webserver_interface=127.8.215.62
--webserver_port=39371
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:42281
--ranger_config_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:02.093637 21903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:02.094002 21903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:02.094168 21903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:02.101696 21903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:02.101871 21903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:02.101986 21903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:02.102080 21903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:02.102169 21903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:02.112519 21903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:42281
--ranger_config_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:02.113955 21903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:02.115343 21903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:02.127669 21909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:02.127727 21911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:02.127770 21908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:02.128636 21903 server_base.cc:1034] running on GCE node
I20260925 05:10:02.129439 21903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:02.130879 21903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:02.132118 21903 hybrid_clock.cc:648] HybridClock initialized: now 1790313002132074 us; error 38 us; skew 500 ppm
I20260925 05:10:02.132470 21903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:02 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313002, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:02.135677 21903 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:02.137354 21903 webserver.cc:460] Webserver started at http://127.8.215.62:39371/ using document root <none> and password file <none>
I20260925 05:10:02.137980 21903 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:02.138082 21903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:10:02 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313002, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:02.203452 21903 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:10:02.207091 21919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:02.208673 21903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:02.208940 21903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "7c7f43e69c4a48ce93a9da761a0a2dde"
format_stamp: "Formatted at 2026-09-25 05:09:58 on dist-test-slave-wpdm"
server_key: "bb52645dea59944ebb5e346e3aa921fe"
server_key_iv: "fe9bdcca41d64ccfb4681e985a2a833b"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:02.209936 21903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:02.226627 21903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:02.230340 21903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:02.230579 21903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:02.249298 21903 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:42281
I20260925 05:10:02.249306 21970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:42281 every 8 connection(s)
I20260925 05:10:02.250864 21903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:02.257422 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 21903
I20260925 05:10:02.257939 21971 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:10:02.258423 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:34793
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:42281
--webserver_port=33479
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:02.261345 21971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Bootstrap starting.
I20260925 05:10:02.280159 21971 log.cc:824] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:02.289671 21971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:02.290202 21971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Bootstrap complete.
I20260925 05:10:02.299278 21971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:10:02.300172 21971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7f43e69c4a48ce93a9da761a0a2dde, State: Initialized, Role: FOLLOWER
I20260925 05:10:02.300860 21971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:10:02.301106 21971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:02.301234 21971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:02.301424 21971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:10:02.305485 21971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:10:02.305996 21971 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7f43e69c4a48ce93a9da761a0a2dde; no voters:
I20260925 05:10:02.306640 21971 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:10:02.306792 21975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:10:02.308007 21975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [term 2 LEADER]: Becoming Leader. State: Replica: 7c7f43e69c4a48ce93a9da761a0a2dde, State: Running, Role: LEADER
I20260925 05:10:02.308630 21975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } }
I20260925 05:10:02.308959 21971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:02.311664 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } } }
I20260925 05:10:02.312202 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:02.311969 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7f43e69c4a48ce93a9da761a0a2dde. Latest consensus state: current_term: 2 leader_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f43e69c4a48ce93a9da761a0a2dde" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 42281 } } }
I20260925 05:10:02.313127 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:02.322646 21982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:02.328676 21982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:02.330231 21982 catalog_manager.cc:1259] Loaded cluster ID: fb965918f23844eaa43f36460fd066f4
I20260925 05:10:02.330430 21982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:02.334564 21982 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:02.335906 21971 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:10:02.336418 21982 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Loaded TSK: 0
I20260925 05:10:02.338057 21982 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:10:02.736235 21973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:02.736771 21973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:02.736982 21973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:02.748978 21973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:02.749233 21973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:02.749475 21973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:10:02.766463 21973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:34793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=33479
--tserver_master_addrs=127.8.215.62:42281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:02.768602 21973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:02.770565 21973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:02.790385 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:02.790405 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:02.799181 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:02.799603 21973 server_base.cc:1034] running on GCE node
I20260925 05:10:02.800338 21973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:02.803421 21973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:02.804668 21973 hybrid_clock.cc:648] HybridClock initialized: now 1790313002804602 us; error 63 us; skew 500 ppm
I20260925 05:10:02.805114 21973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:02 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313002, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:02.808794 21973 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:10:02.810712 21973 webserver.cc:460] Webserver started at http://127.8.215.1:33479/ using document root <none> and password file <none>
I20260925 05:10:02.811456 21973 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:02.811575 21973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:10:02 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313002, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:02.847851 21973 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:10:02.851902 22019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:02.853978 21973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 05:10:02.854252 21973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "aad512203eb34ca0a93207adfcea9953"
format_stamp: "Formatted at 2026-09-25 05:09:59 on dist-test-slave-wpdm"
server_key: "831a888bac306a4405a3bc53929380e7"
server_key_iv: "3ec49c9ea5044f94447e08023a64cc95"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:02.855808 21973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:02.879009 21973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:02.883978 21973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:02.884287 21973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:02.885535 21973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:02.887089 21973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:02.887213 21973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:02.887377 21973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:02.887475 21973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:02.975116 21973 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:34793
I20260925 05:10:02.975164 22132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:34793 every 8 connection(s)
I20260925 05:10:02.976696 21973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:02.986126 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 21973
I20260925 05:10:02.986910 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:46445
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:42281
--webserver_port=39321
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:02 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313002, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:03.012435 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:02.982649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42281 (local address 127.8.215.1:59283)
0925 05:10:02.983495 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:02.983546 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:02.986164 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:02.990473 (+ 4309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:02.990482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:02.990948 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:02.991503 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:02.991521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:02.993889 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:02.993895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:02.995293 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:02.995314 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:02.996899 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:02.996909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:02.997238 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:02.998061 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:02.998096 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:03.001129 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:03.006450 (+ 5321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:03.006464 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:03.006485 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:03.007219 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:03.007809 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:03.007817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:03.007823 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:03.008123 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:03.009274 (+ 1151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:03.009283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:03.009709 (+ 426us) client_negotiation.cc:769] Sending connection context
0925 05:10:03.010032 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 05:10:03.010758 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":748,"threads_started":1}
I20260925 05:10:03.015897 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:02.983850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:59283 (local address 127.8.215.62:42281)
0925 05:10:02.988116 (+ 4266us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:02.988135 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:02.988185 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:10:02.988647 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:02.988667 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:02.989251 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:02.989706 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:02.991820 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:02.993703 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:02.995533 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:02.996609 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:02.996851 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:03.001527 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:03.001540 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:03.001564 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:03.001628 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:03.006245 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:03.007429 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:03.007441 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:03.007453 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:03.007663 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:03.008269 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:03.008276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:03.008283 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:03.009105 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:03.013202 (+ 4097us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:03.013544 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 05:10:03.014072 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3843,"thread_start_us":3612,"threads_started":1}
I20260925 05:10:03.019623 22133 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42281
I20260925 05:10:03.019972 22133 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:03.020895 22133 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
W20260925 05:10:03.024667 22133 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:42281 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:10:03.024933 22133 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:03.025231 22133 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:10:03.026587 22133 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:03.026861 22133 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
W20260925 05:10:03.027966 22133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:10:03.408178 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:03.408586 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:03.408792 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:03.418764 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:03.418970 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:03.419189 22137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:10:03.431521 22137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:46445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=39321
--tserver_master_addrs=127.8.215.62:42281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:03.433058 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:03.434410 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:03.449839 22151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:03.449862 22152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:03.450650 22154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:03.450733 22137 server_base.cc:1034] running on GCE node
I20260925 05:10:03.451473 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:03.452526 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:03.453809 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790313003453730 us; error 66 us; skew 500 ppm
I20260925 05:10:03.454160 22137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:03 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313003, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:03.457444 22137 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:10:03.474087 22137 webserver.cc:460] Webserver started at http://127.8.215.2:39321/ using document root <none> and password file <none>
I20260925 05:10:03.475299 22137 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:03.475483 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:10:03 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313003, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:03.515122 22137 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.009s sys 0.008s
I20260925 05:10:03.521364 22162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:03.523327 22137 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260925 05:10:03.523728 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "7d724666de584da08f17dd1191b22ac6"
format_stamp: "Formatted at 2026-09-25 05:10:00 on dist-test-slave-wpdm"
server_key: "be515d9c500409b0354b64de79fae3db"
server_key_iv: "e901c886b0d2c3f15f16aecac01d45a5"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:03.525182 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:03.592360 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:03.597783 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:03.598104 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:03.599359 22137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:03.601439 22137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:03.601557 22137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:03.601728 22137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:03.601810 22137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:10:03 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790313003, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:03.671680 22137 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:46445
I20260925 05:10:03.671727 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:46445 every 8 connection(s)
I20260925 05:10:03.673354 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:03.678023 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22137
I20260925 05:10:03.678933 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.215.20:37007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:37921
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:42281
--webserver_port=44719
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:03 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313003, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:03.697695 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:03.676796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42281 (local address 127.8.215.2:36129)
0925 05:10:03.677599 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:03.677652 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:03.679533 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:03.680343 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:03.680351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:03.680899 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:03.681545 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:03.681563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:03.682678 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:03.682685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:03.684706 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:03.684759 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:03.686282 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:03.686289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:03.686545 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:03.687280 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:03.687304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:03.689836 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:03.693234 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:03.693243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:03.693261 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:03.693681 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:03.694069 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:03.694073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:03.694077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:03.694306 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:03.694857 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:03.694865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:03.695310 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 05:10:03.695644 (+ 334us) client_negotiation.cc:240] Negotiation successful
0925 05:10:03.696224 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":337,"threads_started":1}
I20260925 05:10:03.698798 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:03.676230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:36129 (local address 127.8.215.62:42281)
0925 05:10:03.676792 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:03.676799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:03.679824 (+ 3025us) server_negotiation.cc:407] Connection header received
0925 05:10:03.679886 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:03.679889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:03.679955 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:03.680083 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:03.681727 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:03.682503 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:03.684911 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:03.685430 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:03.686211 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:03.690087 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:03.690096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:03.690102 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:03.690153 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:03.693069 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:03.693814 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:03.693820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:03.693823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:03.693952 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:03.694433 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:03.694442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:03.694444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:03.694720 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:03.694884 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:03.698193 (+ 3309us) server_negotiation.cc:299] Negotiation successful
0925 05:10:03.698429 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":245,"threads_started":1}
I20260925 05:10:03.701020 22277 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42281
I20260925 05:10:03.701331 22277 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:03.702184 22277 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:10:03.704295 21936 ts_manager.cc:194] Registered new tserver with Master: 7d724666de584da08f17dd1191b22ac6 (127.8.215.2:46445)
I20260925 05:10:03.706475 21936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:36129
I20260925 05:10:04.028147 22133 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:04.028611 22133 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:10:04.030355 21936 ts_manager.cc:194] Registered new tserver with Master: aad512203eb34ca0a93207adfcea9953 (127.8.215.1:34793)
I20260925 05:10:04.031616 21936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:59283
Sep 25 05:10:04 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313003, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
W20260925 05:10:04.179257 22281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:04.179765 22281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:04.179942 22281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:04.190846 22281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:04.191097 22281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:04.191366 22281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:10:04.208047 22281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.215.20:37007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:37921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=44719
--tserver_master_addrs=127.8.215.62:42281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:04.210089 22281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:04.212044 22281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:04.249157 22287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:04.252071 22288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:04.253059 22281 server_base.cc:1034] running on GCE node
W20260925 05:10:04.260142 22291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:04.261214 22281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:04.270728 22281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:04.272184 22281 hybrid_clock.cc:648] HybridClock initialized: now 1790313004272102 us; error 89 us; skew 500 ppm
I20260925 05:10:04.272663 22281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:04 dist-test-slave-wpdm krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313004, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:04.277184 22281 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:10:04.298059 22281 webserver.cc:460] Webserver started at http://127.8.215.3:44719/ using document root <none> and password file <none>
I20260925 05:10:04.299324 22281 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:04.299556 22281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:10:04 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313004, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:04.342630 22281 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 05:10:04.353294 22298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:04.355405 22281 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.004s
I20260925 05:10:04.355677 22281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a"
format_stamp: "Formatted at 2026-09-25 05:10:01 on dist-test-slave-wpdm"
server_key: "33707e74023533a21a358608ecc4fbcf"
server_key_iv: "91c182d0602101672900414eab9aff01"
server_key_version: "kuduclusterkey@0"
I20260925 05:10:04.357295 22281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:04.403585 22281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:04.409209 22281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:04.409693 22281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:04.425207 22281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 25 05:10:04 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313003, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for HTTP/127.8.215.20@KRBTEST.COM
I20260925 05:10:04.433575 22281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:04.433773 22281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:04.433916 22281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:04.434026 22281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:04.561056 22281 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:37921
I20260925 05:10:04.561057 22411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:37921 every 8 connection(s)
I20260925 05:10:04.563143 22281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:04.564666 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22281
I20260925 05:10:04.621234 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.565818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.8.215.62:42281)
0925 05:10:04.593901 (+ 28083us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.593911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.594640 (+ 729us) server_negotiation.cc:407] Connection header received
0925 05:10:04.594894 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.594900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.595005 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.595143 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:04.604923 (+ 9780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.606168 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.608298 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.609055 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.609191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.613128 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:04.613136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:04.613141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:04.613192 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:04.616884 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.618115 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.618120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.618123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.618263 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.618701 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.618706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.618709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.619030 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:04.619260 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.620590 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.620852 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27905,"thread_start_us":27771,"threads_started":1}
Sep 25 05:10:04 dist-test-slave-wpdm krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313004, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:04.630872 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.580972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:56519 (local address 127.8.215.62:42281)
0925 05:10:04.605257 (+ 24285us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.605263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.608315 (+ 3052us) server_negotiation.cc:407] Connection header received
0925 05:10:04.608847 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.608852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.608993 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.609082 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:04.611660 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.612904 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.615344 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.616089 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.616191 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.623536 (+ 7345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:04.623543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:04.623548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:04.623586 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:04.627050 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.628002 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.628008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.628011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.628144 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.628751 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.628756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.628760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.629092 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:04.629208 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.630271 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.630504 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24112,"thread_start_us":23917,"threads_started":1}
I20260925 05:10:04.633467 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.580208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:42281 (local address 127.8.215.3:56519)
0925 05:10:04.605912 (+ 25704us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:04.605978 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:04.608462 (+ 2484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:04.609805 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:04.609819 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:04.610548 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:04.611426 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.611453 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.613119 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.613124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.615160 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.615186 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.616602 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.616609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.616954 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.620118 (+ 3164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:04.620150 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:04.623226 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:04.627223 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:04.627233 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:04.627254 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:04.627860 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:04.628312 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:04.628316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:04.628320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:04.628580 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:04.629381 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:04.629390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:04.629840 (+ 450us) client_negotiation.cc:769] Sending connection context
0925 05:10:04.631435 (+ 1595us) client_negotiation.cc:240] Negotiation successful
0925 05:10:04.632050 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25329,"thread_start_us":25057,"threads_started":1}
I20260925 05:10:04.635093 22412 heartbeater.cc:347] Connected to a master server at 127.8.215.62:42281
I20260925 05:10:04.635473 22412 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:04.636436 22412 heartbeater.cc:511] Master 127.8.215.62:42281 requested a full tablet report, sending...
I20260925 05:10:04.638101 21936 ts_manager.cc:194] Registered new tserver with Master: b2d0ccaf7bf14d0dabeabc8852aaac0a (127.8.215.3:37921)
I20260925 05:10:04.638983 21936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:56519
I20260925 05:10:04.649536 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:04.667718 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.653064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.8.215.62:42281)
0925 05:10:04.653299 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.653308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.653861 (+ 553us) server_negotiation.cc:407] Connection header received
0925 05:10:04.654086 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.654091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.654175 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.654281 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:04.655778 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.656498 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.657935 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.658373 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.658519 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.661468 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:04.661473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:04.661476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:04.661509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:04.664492 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.665495 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.665503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.665507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.665660 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.666068 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.666073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.666077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.666381 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:04.666639 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.667034 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.667327 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:10:04.673205 21936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:04.683684 21936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:04.711335 22277 heartbeater.cc:503] Master 127.8.215.62:42281 was elected leader, sending a full tablet report...
I20260925 05:10:04.728876 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.707887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:34793 (local address 127.0.0.1:57994)
0925 05:10:04.717374 (+ 9487us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:04.717418 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:04.717650 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:04.718755 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:04.718767 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:04.718811 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:04.719338 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.719355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.722073 (+ 2718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.722081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.725277 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.725298 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.727836 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.727842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.728083 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.728146 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 05:10:04.728285 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 05:10:04.728483 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9231,"thread_start_us":9081,"threads_started":1}
I20260925 05:10:04.729651 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.708159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.8.215.1:34793)
0925 05:10:04.717306 (+ 9147us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.717316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.717545 (+ 229us) server_negotiation.cc:407] Connection header received
0925 05:10:04.717902 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.717910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.718203 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.718398 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:04.719604 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.721847 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.725532 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.727289 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.727416 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.727678 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.729138 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.729313 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8939,"thread_start_us":8771,"threads_started":1}
I20260925 05:10:04.733372 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.717987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37921 (local address 127.0.0.1:46832)
0925 05:10:04.725585 (+ 7598us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:04.725610 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:04.725730 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:04.726477 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:04.726484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:04.726514 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:04.727030 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.727046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.729585 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.729590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.731305 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.731318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.732743 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.732752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.732895 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.732919 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:10:04.732968 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:10:04.733055 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7459,"thread_start_us":7307,"threads_started":1}
I20260925 05:10:04.734409 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.718354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.8.215.3:37921)
0925 05:10:04.725237 (+ 6883us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.725246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.725849 (+ 603us) server_negotiation.cc:407] Connection header received
0925 05:10:04.725957 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.725965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.726260 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.727270 (+ 1010us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:04.727570 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.729404 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.731468 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.732460 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.733754 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.733935 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.733990 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.734103 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6677,"thread_start_us":6523,"threads_started":1}
I20260925 05:10:04.737716 22068 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0dec902d2c40daab3ad142ed5b94a0 (DEFAULT_TABLE table=test-table [id=9da46f929cf14789a7289e5b6cb28e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:04.738487 22347 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0dec902d2c40daab3ad142ed5b94a0 (DEFAULT_TABLE table=test-table [id=9da46f929cf14789a7289e5b6cb28e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:04.739378 22068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0dec902d2c40daab3ad142ed5b94a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:04.740430 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0dec902d2c40daab3ad142ed5b94a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:04.748739 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.732307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:46445 (local address 127.0.0.1:60604)
0925 05:10:04.732537 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:04.732558 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:04.732664 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:04.741493 (+ 8829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:04.741499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:04.741532 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:04.741753 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.741767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.744060 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.744066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.746058 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.746076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.747955 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.747962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.748148 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.748190 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:10:04.748277 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:10:04.748430 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 05:10:04.750116 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.726016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.8.215.2:46445)
0925 05:10:04.740592 (+ 14576us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.740602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.740635 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:10:04.740776 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.740785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.741041 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.741950 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:04.742207 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.743862 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.746287 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.747768 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.749292 (+ 1524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.749523 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.749597 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.749779 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14382,"thread_start_us":15187,"threads_started":1}
I20260925 05:10:04.755086 22212 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0dec902d2c40daab3ad142ed5b94a0 (DEFAULT_TABLE table=test-table [id=9da46f929cf14789a7289e5b6cb28e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:04.757051 22212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0dec902d2c40daab3ad142ed5b94a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:04.764091 22439 tablet_bootstrap.cc:492] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Bootstrap starting.
I20260925 05:10:04.764891 22440 tablet_bootstrap.cc:492] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Bootstrap starting.
I20260925 05:10:04.767764 22439 tablet_bootstrap.cc:654] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:04.768394 22440 tablet_bootstrap.cc:654] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:04.768937 22439 log.cc:824] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:04.769690 22440 log.cc:824] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:04.771657 22439 tablet_bootstrap.cc:492] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: No bootstrap required, opened a new log
I20260925 05:10:04.771960 22439 ts_tablet_manager.cc:1397] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 05:10:04.774416 22443 tablet_bootstrap.cc:492] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Bootstrap starting.
I20260925 05:10:04.774883 22440 tablet_bootstrap.cc:492] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: No bootstrap required, opened a new log
I20260925 05:10:04.775210 22440 ts_tablet_manager.cc:1397] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260925 05:10:04.777541 22443 tablet_bootstrap.cc:654] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:04.778517 22443 log.cc:824] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:04.781538 22443 tablet_bootstrap.cc:492] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: No bootstrap required, opened a new log
I20260925 05:10:04.781808 22443 ts_tablet_manager.cc:1397] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:10:04.782908 22439 raft_consensus.cc:348] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.783373 22439 raft_consensus.cc:374] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:04.783505 22439 raft_consensus.cc:729] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad512203eb34ca0a93207adfcea9953, State: Initialized, Role: FOLLOWER
I20260925 05:10:04.784195 22439 consensus_queue.cc:260] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.785434 22133 heartbeater.cc:503] Master 127.8.215.62:42281 was elected leader, sending a full tablet report...
I20260925 05:10:04.786062 22439 ts_tablet_manager.cc:1428] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 05:10:04.785778 22440 raft_consensus.cc:348] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.786213 22440 raft_consensus.cc:374] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:04.786314 22440 raft_consensus.cc:729] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d0ccaf7bf14d0dabeabc8852aaac0a, State: Initialized, Role: FOLLOWER
I20260925 05:10:04.786947 22440 consensus_queue.cc:260] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.788226 22412 heartbeater.cc:503] Master 127.8.215.62:42281 was elected leader, sending a full tablet report...
I20260925 05:10:04.788756 22440 ts_tablet_manager.cc:1428] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 05:10:04.792934 22443 raft_consensus.cc:348] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.793350 22443 raft_consensus.cc:374] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:04.793449 22443 raft_consensus.cc:729] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d724666de584da08f17dd1191b22ac6, State: Initialized, Role: FOLLOWER
I20260925 05:10:04.793994 22443 consensus_queue.cc:260] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.795374 22443 ts_tablet_manager.cc:1428] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 05:10:04.809473 22445 raft_consensus.cc:482] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:04.809832 22445 raft_consensus.cc:504] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.811636 22445 leader_election.cc:288] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2d0ccaf7bf14d0dabeabc8852aaac0a (127.8.215.3:37921), 7d724666de584da08f17dd1191b22ac6 (127.8.215.2:46445)
I20260925 05:10:04.819725 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.812122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:51759 (local address 127.8.215.3:37921)
0925 05:10:04.812435 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.812440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.812929 (+ 489us) server_negotiation.cc:407] Connection header received
0925 05:10:04.813213 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.813223 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.813315 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.813397 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:04.814932 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.815791 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.817826 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.818668 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.818835 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.818967 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.819277 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.819429 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260925 05:10:04.819721 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.813174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:42783 (local address 127.8.215.2:46445)
0925 05:10:04.813470 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.813477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.813935 (+ 458us) server_negotiation.cc:407] Connection header received
0925 05:10:04.814264 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.814269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.814419 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.814530 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:04.815163 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.815839 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.817900 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.818663 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.818840 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.818955 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.819356 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.819497 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 05:10:04.819796 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.812012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37921 (local address 127.8.215.1:51759)
0925 05:10:04.812824 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:04.812845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:04.812985 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:04.813621 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:04.813626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:04.813655 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:04.814140 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.814171 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.815930 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.815938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.817660 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.817672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.818915 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.818922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.819097 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.819135 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:10:04.819211 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 05:10:04.819343 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":30,"thread_start_us":446,"threads_started":1}
I20260925 05:10:04.819912 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.812970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:46445 (local address 127.8.215.1:42783)
0925 05:10:04.813771 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:04.813804 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:04.814082 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:04.814550 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:04.814555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:04.814582 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:04.814983 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.814999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.816027 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.816031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.817720 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:04.817756 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.818968 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.818975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:04.819164 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.819214 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:10:04.819319 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 05:10:04.819444 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":20,"thread_start_us":423,"threads_started":1}
I20260925 05:10:04.820961 22367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0dec902d2c40daab3ad142ed5b94a0" candidate_uuid: "aad512203eb34ca0a93207adfcea9953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" is_pre_election: true
I20260925 05:10:04.820950 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0dec902d2c40daab3ad142ed5b94a0" candidate_uuid: "aad512203eb34ca0a93207adfcea9953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d724666de584da08f17dd1191b22ac6" is_pre_election: true
I20260925 05:10:04.821447 22367 raft_consensus.cc:2454] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad512203eb34ca0a93207adfcea9953 in term 0.
I20260925 05:10:04.821506 22232 raft_consensus.cc:2454] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad512203eb34ca0a93207adfcea9953 in term 0.
I20260925 05:10:04.822211 22022 leader_election.cc:302] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad512203eb34ca0a93207adfcea9953, b2d0ccaf7bf14d0dabeabc8852aaac0a; no voters:
I20260925 05:10:04.822572 22445 raft_consensus.cc:2781] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:04.822785 22445 raft_consensus.cc:482] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:04.822921 22445 raft_consensus.cc:3037] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:04.825150 22445 raft_consensus.cc:504] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.825719 22445 leader_election.cc:288] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [CANDIDATE]: Term 1 election: Requested vote from peers b2d0ccaf7bf14d0dabeabc8852aaac0a (127.8.215.3:37921), 7d724666de584da08f17dd1191b22ac6 (127.8.215.2:46445)
I20260925 05:10:04.826459 22367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0dec902d2c40daab3ad142ed5b94a0" candidate_uuid: "aad512203eb34ca0a93207adfcea9953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a"
I20260925 05:10:04.826459 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0dec902d2c40daab3ad142ed5b94a0" candidate_uuid: "aad512203eb34ca0a93207adfcea9953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d724666de584da08f17dd1191b22ac6"
I20260925 05:10:04.826674 22232 raft_consensus.cc:3037] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:04.826673 22367 raft_consensus.cc:3037] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:04.829926 22367 raft_consensus.cc:2454] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad512203eb34ca0a93207adfcea9953 in term 1.
I20260925 05:10:04.829959 22232 raft_consensus.cc:2454] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad512203eb34ca0a93207adfcea9953 in term 1.
I20260925 05:10:04.830340 22022 leader_election.cc:302] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d724666de584da08f17dd1191b22ac6, aad512203eb34ca0a93207adfcea9953; no voters:
I20260925 05:10:04.830801 22445 raft_consensus.cc:2781] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:04.831310 22445 raft_consensus.cc:686] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 1 LEADER]: Becoming Leader. State: Replica: aad512203eb34ca0a93207adfcea9953, State: Running, Role: LEADER
I20260925 05:10:04.831816 22445 consensus_queue.cc:237] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } }
I20260925 05:10:04.836305 21936 catalog_manager.cc:5495] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad512203eb34ca0a93207adfcea9953 (127.8.215.1). New cstate: current_term: 1 leader_uuid: "aad512203eb34ca0a93207adfcea9953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad512203eb34ca0a93207adfcea9953" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 34793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 } health_report { overall_health: UNKNOWN } } }
W20260925 05:10:04.843784 22415 tablet.cc:2369] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:04.862844 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.857047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.8.215.1:34793)
0925 05:10:04.857265 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.857271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.857291 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:10:04.857522 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.857526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.857630 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.857784 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:04.858161 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.858935 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.860340 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.861010 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.861159 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.861737 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:04.862285 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:04.862445 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.862567 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.862636 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:10:04.871528 22232 raft_consensus.cc:1261] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 1 FOLLOWER]: Refusing update from remote peer aad512203eb34ca0a93207adfcea9953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:04.871528 22367 raft_consensus.cc:1261] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 1 FOLLOWER]: Refusing update from remote peer aad512203eb34ca0a93207adfcea9953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:04.872536 22450 consensus_queue.cc:1035] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d0ccaf7bf14d0dabeabc8852aaac0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:04.873030 22445 consensus_queue.cc:1035] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d724666de584da08f17dd1191b22ac6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:04.885071 22453 mvcc.cc:205] Tried to move back new op lower bound from 7333122067942731776 to 7333122067795247104. Current Snapshot: MvccSnapshot[applied={T|T < 7333122067942731776}]
I20260925 05:10:04.898175 21936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:04.898396 21936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:04.901897 21936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c7f43e69c4a48ce93a9da761a0a2dde: Sending DeleteTablet for 3 replicas of tablet 0b0dec902d2c40daab3ad142ed5b94a0
I20260925 05:10:04.903023 22068 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0dec902d2c40daab3ad142ed5b94a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:04 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:57994
I20260925 05:10:04.903128 22347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0dec902d2c40daab3ad142ed5b94a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:04 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:46832
I20260925 05:10:04.903218 22212 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0dec902d2c40daab3ad142ed5b94a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:04 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:60604
I20260925 05:10:04.911592 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 21973
I20260925 05:10:04.913848 22465 tablet_replica.cc:330] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: stopping tablet replica
I20260925 05:10:04.914403 22465 raft_consensus.cc:2229] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:04.915128 22465 raft_consensus.cc:2258] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:04.915961 22463 tablet_replica.cc:330] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: stopping tablet replica
I20260925 05:10:04.916518 22463 raft_consensus.cc:2229] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:04.917220 22463 raft_consensus.cc:2258] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:04.917949 22465 ts_tablet_manager.cc:1905] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:04.919590 22463 ts_tablet_manager.cc:1905] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:04.921972 22464 tablet_replica.cc:330] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: stopping tablet replica
I20260925 05:10:04.922497 22464 raft_consensus.cc:2229] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:04.923086 22464 raft_consensus.cc:2258] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:04.925354 22464 ts_tablet_manager.cc:1905] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:04.926955 22465 ts_tablet_manager.cc:1918] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:04.927171 22465 log.cc:1196] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/0b0dec902d2c40daab3ad142ed5b94a0
I20260925 05:10:04.928180 22465 ts_tablet_manager.cc:1939] T 0b0dec902d2c40daab3ad142ed5b94a0 P 7d724666de584da08f17dd1191b22ac6: Deleting consensus metadata
I20260925 05:10:04.930172 21922 catalog_manager.cc:4841] TS 7d724666de584da08f17dd1191b22ac6 (127.8.215.2:46445): tablet 0b0dec902d2c40daab3ad142ed5b94a0 (table test-table [id=9da46f929cf14789a7289e5b6cb28e90]) successfully deleted
I20260925 05:10:04.932610 22464 ts_tablet_manager.cc:1918] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:04.932891 22464 log.cc:1196] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/0b0dec902d2c40daab3ad142ed5b94a0
I20260925 05:10:04.933240 22463 ts_tablet_manager.cc:1918] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:04.933457 22463 log.cc:1196] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/0b0dec902d2c40daab3ad142ed5b94a0
I20260925 05:10:04.933485 22464 ts_tablet_manager.cc:1939] T 0b0dec902d2c40daab3ad142ed5b94a0 P b2d0ccaf7bf14d0dabeabc8852aaac0a: Deleting consensus metadata
I20260925 05:10:04.934119 22463 ts_tablet_manager.cc:1939] T 0b0dec902d2c40daab3ad142ed5b94a0 P aad512203eb34ca0a93207adfcea9953: Deleting consensus metadata
I20260925 05:10:04.935354 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:04.913678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.8.215.1:34793)
0925 05:10:04.913910 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:04.913918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:04.913944 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:10:04.914018 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:04.914026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:04.914124 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:04.914258 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:04.917476 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.919739 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.922414 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:04.923094 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:04.923238 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:04.926690 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:04.926706 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:04.926727 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:04.926776 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:04.931036 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.932006 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.932015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.932019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.932192 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:04.932618 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:04.932623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:04.932627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:04.933148 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:04.933459 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:04.934758 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0925 05:10:04.934998 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:10:04.935359 21922 catalog_manager.cc:4841] TS b2d0ccaf7bf14d0dabeabc8852aaac0a (127.8.215.3:37921): tablet 0b0dec902d2c40daab3ad142ed5b94a0 (table test-table [id=9da46f929cf14789a7289e5b6cb28e90]) successfully deleted
I20260925 05:10:04.936242 22128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:04.936602 21922 catalog_manager.cc:4841] TS aad512203eb34ca0a93207adfcea9953 (127.8.215.1:34793): tablet 0b0dec902d2c40daab3ad142ed5b94a0 (table test-table [id=9da46f929cf14789a7289e5b6cb28e90]) successfully deleted
I20260925 05:10:05.036015 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 21973
I20260925 05:10:05.054235 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22137
I20260925 05:10:05.067107 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:05.055070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.8.215.2:46445)
0925 05:10:05.055273 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:05.055278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:05.055297 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:10:05.055388 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:05.055392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:05.055462 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:05.055545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:05.056853 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:05.057563 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:05.058759 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:05.059154 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:05.059245 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:05.061706 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:05.061719 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:05.061772 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:05.061815 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:05.064625 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:05.065335 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:05.065342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:05.065345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:05.065478 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:05.065781 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:05.065784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:05.065786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:05.066114 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:05.066345 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:05.066565 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 05:10:05.066725 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 05:10:05.067837 22272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:05.158670 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22137
I20260925 05:10:05.176112 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22281
I20260925 05:10:05.189231 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:05.176931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.8.215.3:37921)
0925 05:10:05.177117 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:05.177121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:05.177140 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:10:05.177284 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:05.177288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:05.177361 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:05.177445 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:05.178783 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:05.179502 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:05.180918 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:05.181311 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:05.181407 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:05.183872 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:05.183887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:05.183905 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:05.183936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:05.186624 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:05.187251 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:05.187257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:05.187269 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:05.187383 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:05.187707 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:05.187710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:05.187712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:05.188093 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:05.188331 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:05.188644 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 05:10:05.188893 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 05:10:05.189981 22407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:05.279350 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22281
I20260925 05:10:05.296407 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 21903
I20260925 05:10:05.297798 21966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:05.410573 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 21903
I20260925 05:10:05.435468 9052 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:10:05.771090 9052 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:10:05.771271 9052 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:10:06.116675 9052 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:10:06.117 UTC [20955] LOG: received smart shutdown request
2026-09-25 05:10:06.120 UTC [20955] LOG: background worker "logical replication launcher" (PID 20964) exited with exit code 1
2026-09-25 05:10:06.120 UTC [20959] LOG: shutting down
2026-09-25 05:10:06.297 UTC [20955] LOG: database system is shut down
2026-09-25T05:10:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54389 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:06 dist-test-slave-wpdm krb5kdc[22478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:06 dist-test-slave-wpdm krb5kdc[22478](info): set up 2 sockets
Sep 25 05:10:06 dist-test-slave-wpdm krb5kdc[22478](info): commencing operation
krb5kdc: starting...
W20260925 05:10:07.549573 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:10:07 dist-test-slave-wpdm krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:10:07.800050 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:46521
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:46521,127.8.215.61:42487,127.8.215.60:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:08.131553 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:08.131937 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:08.132114 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:08.139405 22491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:08.139585 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:08.139699 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:08.139812 22491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:08.139899 22491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:08.149818 22491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:46521,127.8.215.61:42487,127.8.215.60:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:08.151250 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:08.152595 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:08.164570 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:08.164537 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:08.164800 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:08.165639 22491 server_base.cc:1034] running on GCE node
I20260925 05:10:08.166371 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:08.167734 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:08.168936 22491 hybrid_clock.cc:648] HybridClock initialized: now 1790313008168869 us; error 40 us; skew 500 ppm
I20260925 05:10:08.169284 22491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:08 dist-test-slave-wpdm krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:08.172439 22491 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:08.174089 22491 webserver.cc:460] Webserver started at http://127.8.215.62:43665/ using document root <none> and password file <none>
I20260925 05:10:08.174831 22491 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:08.174961 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:08.175305 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:08.178544 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "95db563521104233b68bef061ea351c8"
format_stamp: "Formatted at 2026-09-25 05:10:08 on dist-test-slave-wpdm"
server_key: "95aa4c26873d18bb08f6877cc426bdb0"
server_key_iv: "aa9b70cbc28093a4d7a03e3f7a4486f7"
server_key_version: "encryptionkey@0"
I20260925 05:10:08.179459 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "95db563521104233b68bef061ea351c8"
format_stamp: "Formatted at 2026-09-25 05:10:08 on dist-test-slave-wpdm"
server_key: "95aa4c26873d18bb08f6877cc426bdb0"
server_key_iv: "aa9b70cbc28093a4d7a03e3f7a4486f7"
server_key_version: "encryptionkey@0"
I20260925 05:10:08.184917 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:10:08.188454 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:08.189949 22491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:08.190136 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "95db563521104233b68bef061ea351c8"
format_stamp: "Formatted at 2026-09-25 05:10:08 on dist-test-slave-wpdm"
server_key: "95aa4c26873d18bb08f6877cc426bdb0"
server_key_iv: "aa9b70cbc28093a4d7a03e3f7a4486f7"
server_key_version: "encryptionkey@0"
I20260925 05:10:08.190332 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:08.227043 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:08.231556 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:08.231885 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:08.251469 22491 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:46521
I20260925 05:10:08.251523 22557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:46521 every 8 connection(s)
I20260925 05:10:08.253008 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:08.253933 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22491
I20260925 05:10:08.254187 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:10:08.254824 9052 external_mini_cluster.cc:1428] Setting key bf80660cad17329122dcad56ee0c979a
I20260925 05:10:08.257262 22558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:08.264108 22558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.215.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.61@KRBTEST.COM" created.
Sep 25 05:10:08 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:08.289604 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.267652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.8.215.62:46521)
0925 05:10:08.269557 (+ 1905us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:08.269570 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:08.269628 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:10:08.271355 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:08.271365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:08.271692 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:08.271936 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:08.273291 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.275416 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.277210 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.277867 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.278002 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.282138 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:08.282148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:08.282161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:08.282194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:08.285123 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.285837 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.285844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.285846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.286015 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.286420 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.286425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.286434 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.286950 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:08.287165 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:08.287708 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 05:10:08.288058 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1668,"spinlock_wait_cycles":21248,"thread_start_us":1423,"threads_started":1}
I20260925 05:10:08.289642 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.267653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:46521 (local address 127.0.0.1:54280)
0925 05:10:08.268806 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.268872 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:08.270924 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:08.271997 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:08.272006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:08.272657 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:08.273067 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.273087 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.275642 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.275647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.277064 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.277078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.278123 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.278129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.278267 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.279037 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:08.279062 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:08.281894 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:08.285267 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.285282 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.285296 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.285682 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.286130 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.286136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.286140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.286307 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.287091 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:08.287103 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:08.287317 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 05:10:08.287569 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 05:10:08.288171 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":141,"thread_start_us":456,"threads_started":1}
I20260925 05:10:08.292750 22558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:10:08.294111 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.293709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47818)
0925 05:10:08.293882 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.293994 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 05:10:08.294570 22508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
I20260925 05:10:08.296356 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.296014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47826)
0925 05:10:08.296160 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.296226 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:10:08.296756 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:42487: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab.
Entry for principal kudu/127.8.215.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab.
I20260925 05:10:08.312968 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:42487
--webserver_interface=127.8.215.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab
--principal=kudu/127.8.215.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:46521,127.8.215.61:42487,127.8.215.60:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:08.346165 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } attempt: 1
I20260925 05:10:08.347383 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.346917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47836)
0925 05:10:08.347117 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.347212 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 05:10:08.348490 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.348034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47844)
0925 05:10:08.348220 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.348316 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:10:08.348711 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:42487: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
I20260925 05:10:08.417004 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } attempt: 2
I20260925 05:10:08.418320 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.417747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47856)
0925 05:10:08.418017 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.418149 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:10:08.419384 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.418958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47868)
0925 05:10:08.419122 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.419223 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:10:08.419610 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:42487: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
I20260925 05:10:08.510891 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } attempt: 3
I20260925 05:10:08.512283 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.511653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47870)
0925 05:10:08.511912 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.512062 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 05:10:08.513576 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.513067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47880)
0925 05:10:08.513267 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.513372 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 05:10:08.513830 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:42487: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
W20260925 05:10:08.650534 22565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:08.650826 22565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:08.650913 22565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 05:10:08.657084 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } attempt: 4
I20260925 05:10:08.658300 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.657702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47882)
0925 05:10:08.658037 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.658145 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260925 05:10:08.658417 22565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:08.658533 22565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:08.658622 22565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:08.658706 22565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:08.658771 22565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:08.659365 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.658971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47886)
0925 05:10:08.659124 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.659225 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 05:10:08.659544 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:42487: Network error: Client connection negotiation failed: client connection to 127.8.215.61:42487: connect: Connection refused (error 111)
I20260925 05:10:08.669082 22565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:46521,127.8.215.61:42487,127.8.215.60:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab
--principal=kudu/127.8.215.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.215.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:08.670336 22565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:08.671576 22565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:08.683128 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:08.683290 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:08.683332 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:08.684423 22565 server_base.cc:1034] running on GCE node
I20260925 05:10:08.685283 22565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:08.686693 22565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:08.687925 22565 hybrid_clock.cc:648] HybridClock initialized: now 1790313008687856 us; error 62 us; skew 500 ppm
I20260925 05:10:08.688251 22565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:08 dist-test-slave-wpdm krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:08.691597 22565 init.cc:377] Logged in from keytab as kudu/127.8.215.61@KRBTEST.COM (short username kudu)
I20260925 05:10:08.693111 22565 webserver.cc:460] Webserver started at http://127.8.215.61:35337/ using document root <none> and password file <none>
I20260925 05:10:08.693852 22565 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:08.693975 22565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:08.694298 22565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:08.697611 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data/instance:
uuid: "4ad14e1d486c431ea518da837108b1f2"
format_stamp: "Formatted at 2026-09-25 05:10:08 on dist-test-slave-wpdm"
server_key: "d9b6e412d6ed4ab05bb6fd1130d48bbb"
server_key_iv: "9c9dc950c923b7b0b56b82a94843f299"
server_key_version: "encryptionkey@0"
I20260925 05:10:08.698470 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal/instance:
uuid: "4ad14e1d486c431ea518da837108b1f2"
format_stamp: "Formatted at 2026-09-25 05:10:08 on dist-test-slave-wpdm"
server_key: "d9b6e412d6ed4ab05bb6fd1130d48bbb"
server_key_iv: "9c9dc950c923b7b0b56b82a94843f299"
server_key_version: "encryptionkey@0"
I20260925 05:10:08.703832 22565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:10:08.707438 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:08.708868 22565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:08.709043 22565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal
uuid: "4ad14e1d486c431ea518da837108b1f2"
format_stamp: "Formatted at 2026-09-25 05:10:08 on dist-test-slave-wpdm"
server_key: "d9b6e412d6ed4ab05bb6fd1130d48bbb"
server_key_iv: "9c9dc950c923b7b0b56b82a94843f299"
server_key_version: "encryptionkey@0"
I20260925 05:10:08.709244 22565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:08.737212 22565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:08.741266 22565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:08.741564 22565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:08.759853 22565 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.61:42487
I20260925 05:10:08.759887 22631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.61:42487 every 8 connection(s)
I20260925 05:10:08.761232 22565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
I20260925 05:10:08.764964 22632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:08.767112 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22565
I20260925 05:10:08.767318 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-1/wal/instance
I20260925 05:10:08.767740 9052 external_mini_cluster.cc:1428] Setting key f39cce38fcc7609a719cd73b1afea191
I20260925 05:10:08.771562 22632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:08 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.215.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.60@KRBTEST.COM" created.
I20260925 05:10:08.793583 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.774365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.8.215.62:46521)
0925 05:10:08.774565 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:08.774573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:08.775977 (+ 1404us) server_negotiation.cc:407] Connection header received
0925 05:10:08.777896 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:08.777905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:08.778003 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:08.778132 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:08.779568 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.780742 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.782283 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.783072 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.783219 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.786689 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:08.786698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:08.786707 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:08.786778 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:08.790277 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.791003 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.791007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.791009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.791125 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.791581 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.791584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.791586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.791862 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:08.791980 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:08.792967 (+ 987us) server_negotiation.cc:299] Negotiation successful
0925 05:10:08.793258 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:10:08.795032 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.774967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:46521 (local address 127.0.0.1:54288)
0925 05:10:08.775796 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.775840 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:08.777608 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:08.778321 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:08.778328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:08.778843 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:08.779385 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.779402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.780902 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.780906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.782130 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.782141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.783284 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.783288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.783468 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.784169 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:08.784198 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:08.786434 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:08.790417 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.790424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.790438 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.790882 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.791256 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.791260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.791263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.791475 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.791998 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:08.792008 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:08.792404 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 05:10:08.792871 (+ 467us) client_negotiation.cc:240] Negotiation successful
0925 05:10:08.793433 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":354,"threads_started":1}
I20260925 05:10:08.796813 22632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:08 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:10:08.813815 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.797816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47900)
0925 05:10:08.797980 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.797995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:08.798083 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:08.799219 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:08.799226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:08.799761 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:08.799981 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.799995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.801446 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.801454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.802761 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.802773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.804147 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.804151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.804321 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.805012 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:08.805030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:08.807379 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:08.810691 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.810699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.810705 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.811266 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.811716 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.811720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.811724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.811922 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.812559 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:08.812567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:08.812804 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 05:10:08.812989 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 05:10:08.813350 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 05:10:08.813835 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.798148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.8.215.61:42487)
0925 05:10:08.798778 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:08.798785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:08.798810 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:10:08.798854 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:08.798860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:08.799042 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:08.799161 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:08.800141 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.801292 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.802909 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.803946 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.804095 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.807614 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:08.807628 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:08.807644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:08.807674 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:08.810550 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.811416 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.811428 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.811430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.811588 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.812027 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.812030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.812032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.812384 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:08.812578 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:08.813148 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 05:10:08.813446 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":489,"thread_start_us":366,"threads_started":1}
I20260925 05:10:08.816632 22632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:10:08.817855 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.817398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:60940)
0925 05:10:08.817565 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.817701 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:10:08.818301 22584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:08.819809 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.819452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:60956)
0925 05:10:08.819620 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.819697 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 05:10:08.820124 22632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab.
Entry for principal kudu/127.8.215.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab.
I20260925 05:10:08.824473 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:34189
--webserver_interface=127.8.215.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab
--principal=kudu/127.8.215.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:46521,127.8.215.61:42487,127.8.215.60:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:08.869444 22632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 1
I20260925 05:10:08.870716 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.870131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:60958)
0925 05:10:08.870381 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.870521 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 05:10:08.871924 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.871420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:60972)
0925 05:10:08.871633 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.871758 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 05:10:08.872207 22632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:08.943867 22632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 2
I20260925 05:10:08.945225 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.944637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:60982)
0925 05:10:08.944915 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.945029 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 05:10:08.946329 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.945932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:60998)
0925 05:10:08.946098 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.946189 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 05:10:08.946625 22632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:08.958791 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } attempt: 5
Sep 25 05:10:08 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:10:08.977176 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.960155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.8.215.61:42487)
0925 05:10:08.960328 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:08.960333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:08.960350 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:10:08.960423 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:08.960427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:08.960489 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:08.960596 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:08.961705 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.962906 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.964678 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.965544 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.966439 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.970367 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:08.970377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:08.970385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:08.970438 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:08.973888 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.974753 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.974761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.974765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.974908 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:08.975346 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:08.975351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:08.975355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:08.975706 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:08.975885 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:08.976501 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 05:10:08.976808 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:10:08.977182 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.959558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47904)
0925 05:10:08.959792 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.959818 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:08.959993 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:08.960679 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:08.960685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:08.961218 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:08.961453 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.961470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.963075 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.963084 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.964505 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:08.964519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:08.965765 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:08.965783 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:08.966040 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:08.966897 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:08.966930 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:08.970148 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:08.974060 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.974068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.974073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.974621 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.975045 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:08.975051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:08.975055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:08.975213 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:08.975962 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:08.975970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:08.976177 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 05:10:08.976383 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 05:10:08.976725 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 05:10:08.978722 22558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:10:08.979925 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.979519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32780)
0925 05:10:08.979696 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.979776 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 05:10:08.981081 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:08.980666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32784)
0925 05:10:08.980888 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:08.980961 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 05:10:08.981328 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:09.032575 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 1
I20260925 05:10:09.033874 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.033404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32796)
0925 05:10:09.033600 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.033710 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 05:10:09.034910 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.034566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32810)
0925 05:10:09.034710 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.034806 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:10:09.035118 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:09.038822 22632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 3
I20260925 05:10:09.039860 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.039366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32814)
0925 05:10:09.039582 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.039683 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 05:10:09.041146 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.040571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32822)
0925 05:10:09.040869 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.040980 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260925 05:10:09.041400 22632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:09.103360 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 2
I20260925 05:10:09.104553 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.104036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32830)
0925 05:10:09.104296 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.104380 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 05:10:09.105769 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.105379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32834)
0925 05:10:09.105551 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.105648 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 05:10:09.106079 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
W20260925 05:10:09.165518 22639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:09.165804 22639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:09.165963 22639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:09.173185 22639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:09.173296 22639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:09.173385 22639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:09.173491 22639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:09.173568 22639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:09.183527 22639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:46521,127.8.215.61:42487,127.8.215.60:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab
--principal=kudu/127.8.215.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.215.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:09.185045 22639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:09.184660 22632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 4
I20260925 05:10:09.186151 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.185589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32844)
0925 05:10:09.185840 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.185941 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 05:10:09.186785 22639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:10:09.187233 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.186861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32856)
0925 05:10:09.187039 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.187114 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 05:10:09.187505 22632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
W20260925 05:10:09.199254 22645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:09.199254 22644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:09.199304 22647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:09.200059 22639 server_base.cc:1034] running on GCE node
I20260925 05:10:09.200905 22639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:09.202361 22639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:09.203615 22639 hybrid_clock.cc:648] HybridClock initialized: now 1790313009203529 us; error 81 us; skew 500 ppm
I20260925 05:10:09.203982 22639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:09.207222 22639 init.cc:377] Logged in from keytab as kudu/127.8.215.60@KRBTEST.COM (short username kudu)
I20260925 05:10:09.208829 22639 webserver.cc:460] Webserver started at http://127.8.215.60:45825/ using document root <none> and password file <none>
I20260925 05:10:09.209556 22639 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:09.209694 22639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:09.209997 22639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:09.212306 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 3
I20260925 05:10:09.213405 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.212945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32872)
0925 05:10:09.213157 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.213249 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 05:10:09.213302 22639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data/instance:
uuid: "2f67c73b9e36480babbafdaf915580de"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-wpdm"
server_key: "83963375a496e93d8ef990c7ac3ba1ef"
server_key_iv: "7de463b767144c132b6c8bf899c59677"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.214229 22639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal/instance:
uuid: "2f67c73b9e36480babbafdaf915580de"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-wpdm"
server_key: "83963375a496e93d8ef990c7ac3ba1ef"
server_key_iv: "7de463b767144c132b6c8bf899c59677"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.214440 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.214023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32874)
0925 05:10:09.214243 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.214315 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 05:10:09.214630 22558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:34189: Network error: Client connection negotiation failed: client connection to 127.8.215.60:34189: connect: Connection refused (error 111)
I20260925 05:10:09.219811 22639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 05:10:09.223282 22654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:09.224648 22639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:09.224879 22639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal
uuid: "2f67c73b9e36480babbafdaf915580de"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-wpdm"
server_key: "83963375a496e93d8ef990c7ac3ba1ef"
server_key_iv: "7de463b767144c132b6c8bf899c59677"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.225092 22639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:09.246217 22639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:09.250439 22639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:09.250736 22639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:09.269337 22639 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.60:34189
I20260925 05:10:09.269436 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.60:34189 every 8 connection(s)
I20260925 05:10:09.270766 22639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
I20260925 05:10:09.274525 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:09.279873 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22639
I20260925 05:10:09.280066 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312801912195-9052-0/minicluster-data/master-2/wal/instance
I20260925 05:10:09.280579 9052 external_mini_cluster.cc:1428] Setting key a9bc195f8ebcc317a4d3baed86118bc5
I20260925 05:10:09.281955 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
I20260925 05:10:09.303246 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.284807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.8.215.62:46521)
0925 05:10:09.285004 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.285012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.285823 (+ 811us) server_negotiation.cc:407] Connection header received
0925 05:10:09.286211 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.286217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.286297 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.286401 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.288146 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.289309 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.291380 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.291985 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.292133 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.296495 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.296503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.296509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.296553 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.299522 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.300421 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.300426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.300428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.300568 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.301095 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.301102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.301108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.301492 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.301675 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.302577 (+ 902us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.302825 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:10:09.304483 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.284802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:46521 (local address 127.0.0.1:54298)
0925 05:10:09.285665 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.285712 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:09.285998 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:09.286480 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:09.286489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:09.287395 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:09.287961 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.287985 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.289465 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.289469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.291211 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.291226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.292193 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.292198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.292371 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.293168 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:09.293210 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:09.296273 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:09.299659 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.299672 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.299687 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.300283 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.300684 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.300689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.300692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.300980 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.301637 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:09.301645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:09.302086 (+ 441us) client_negotiation.cc:769] Sending connection context
0925 05:10:09.302473 (+ 387us) client_negotiation.cc:240] Negotiation successful
0925 05:10:09.302982 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"spinlock_wait_cycles":25344,"thread_start_us":446,"threads_started":1}
I20260925 05:10:09.304503 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.282649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.8.215.60:34189)
0925 05:10:09.283553 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.283565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.283606 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:10:09.285629 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.285651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.286003 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.286260 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.288138 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.289820 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.291586 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.292423 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.292519 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.297477 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.297490 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.297510 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.297557 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.300822 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.301658 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.301667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.301669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.301813 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.302179 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.302184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.302187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.302600 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.302849 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.303234 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.303478 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":682,"thread_start_us":454,"threads_started":1}
I20260925 05:10:09.306274 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:09.320477 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.307098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:42487 (local address 127.0.0.1:47918)
0925 05:10:09.307249 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.307268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:09.307386 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:09.307709 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:09.307713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:09.308175 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:09.308366 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.308376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.309548 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.309556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.310596 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.310608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.311363 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.311370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.311562 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.312134 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:09.312158 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:09.314531 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:09.318004 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.318012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.318016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.318425 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.318847 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.318852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.318856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.319005 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.319496 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:09.319501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:09.319664 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 05:10:09.319807 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 05:10:09.320090 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 05:10:09.320502 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.307071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.8.215.61:42487)
0925 05:10:09.307270 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.307279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.307323 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:10:09.307523 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.307527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.307591 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.307689 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.308539 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.309402 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.310741 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.311222 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.311315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.314728 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.314733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.314737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.314769 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.317604 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.318567 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.318573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.318577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.318736 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.319111 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.319117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.319121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.319381 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.319539 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.319926 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.320145 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:10:09.321540 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:10:09.323433 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.308017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.8.215.62:46521)
0925 05:10:09.308185 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.308190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.308207 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:10:09.308333 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.308338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.308406 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.308501 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.309979 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.310755 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.312446 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.313015 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.313110 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.317624 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.317629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.317632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.317664 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.320270 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.321086 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.321096 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.321099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.321245 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.321658 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.321662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.321664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.321972 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.322275 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.323034 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.323210 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:10:09.336429 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.322226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32904)
0925 05:10:09.322382 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.322401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:09.322501 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:09.322943 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:09.322948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:09.323344 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:09.323470 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.323502 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.324616 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.324620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.325898 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.325910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.326902 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.326906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.327056 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.327586 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:09.327609 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:09.330198 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:09.333969 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.333972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.333975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.334478 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.334935 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.334940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.334943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.335100 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.335614 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:09.335620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:09.335782 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 05:10:09.335944 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 05:10:09.336158 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:10:09.336709 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.322479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.8.215.60:34189)
0925 05:10:09.322636 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.322643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.322661 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:10:09.322710 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.322715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.322788 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.322883 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.323669 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.324477 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.326028 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.326769 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.326897 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.330376 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.330380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.330383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.330437 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.333831 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.334667 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.334671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.334673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.334799 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.335216 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.335219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.335221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.335487 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.335595 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.336067 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.336330 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43}
I20260925 05:10:09.341991 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.325224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.8.215.61:42487)
0925 05:10:09.325401 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.325407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.325425 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:10:09.325520 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.325525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.325599 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.325686 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.327201 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.328277 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.329703 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.330343 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.330438 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.335871 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.335876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.335879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.335910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.339129 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.340047 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.340051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.340053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.340191 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.340659 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.340662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.340664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.341013 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.341224 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.341590 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.341768 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:10:09.343006 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: Bootstrap starting.
I20260925 05:10:09.347352 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:09.347580 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:09.348587 22706 log.cc:824] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:09.351924 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: No bootstrap required, opened a new log
I20260925 05:10:09.366189 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.366849 22706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:09.367059 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f67c73b9e36480babbafdaf915580de, State: Initialized, Role: FOLLOWER
I20260925 05:10:09.368013 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.377660 22719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.378247 22719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.379216 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:09.382184 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.353771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.8.215.62:46521)
0925 05:10:09.353953 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.353961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.362508 (+ 8547us) server_negotiation.cc:407] Connection header received
0925 05:10:09.363164 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.363174 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.363284 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.363377 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.366600 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.367758 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.370129 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.370782 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.370884 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.375118 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.375127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.375134 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.375181 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.378805 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.379702 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.379707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.379710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.379870 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.380360 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.380368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.380373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.380769 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.381030 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.381599 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.381844 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:10:09.387931 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.353967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.8.215.61:42487)
0925 05:10:09.354146 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.354152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.365908 (+ 11756us) server_negotiation.cc:407] Connection header received
0925 05:10:09.366860 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.366869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.366961 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.367044 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.369953 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.371426 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.374311 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.375221 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.375345 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.379193 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.379201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.379209 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.379257 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.384092 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.385381 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.385390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.385396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.385568 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.386011 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.386016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.386019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.386384 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.386647 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.387318 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.387564 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:10:09.387946 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.353535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.8.215.60:34189)
0925 05:10:09.353759 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.353765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.366607 (+ 12842us) server_negotiation.cc:407] Connection header received
0925 05:10:09.366888 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.366895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.366978 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.367054 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.369329 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.370489 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.372811 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.373658 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.373766 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.379505 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.379511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.379516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.379565 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.384145 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.385268 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.385277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.385287 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.385516 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.386014 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.386019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.386022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.386435 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.386667 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.387333 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.387555 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:10:09.391840 22558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 4
W20260925 05:10:09.395030 22730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:09.395326 22730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
I20260925 05:10:09.391702 22639 master_runner.cc:186] Error getting master registration for 127.8.215.62:46521: OK, instance_id { permanent_uuid: "95db563521104233b68bef061ea351c8" instance_seqno: 1790313008245983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:10:09.403503 22639 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:10:09.408490 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.392437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.8.215.60:34189)
0925 05:10:09.392661 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.392672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.392692 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:10:09.392874 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.392883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.392993 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.393102 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.394205 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.395484 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.397248 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.397998 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.398137 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.401952 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.401957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.401960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.401993 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.405877 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.406518 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.406523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.406527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.406656 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.407039 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.407043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.407047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.407334 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.407491 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.407897 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.408125 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 05:10:09.408509 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.392341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32920)
0925 05:10:09.392568 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.392587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:09.392690 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:09.393280 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:09.393285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:09.393800 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:09.394022 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.394034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.395630 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.395638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.397074 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.397089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.398385 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.398390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.398596 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.399308 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:09.399332 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:09.401741 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:09.406004 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.406011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.406015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.406391 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.406768 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.406773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.406777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.406921 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.407467 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:09.407472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:09.407624 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 05:10:09.407776 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 05:10:09.408113 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 05:10:09.413385 22558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: Bootstrap starting.
I20260925 05:10:09.416529 22558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:09.417606 22558 log.cc:824] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:09.420706 22558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: No bootstrap required, opened a new log
I20260925 05:10:09.428061 22558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.428471 22558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:09.428607 22558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95db563521104233b68bef061ea351c8, State: Initialized, Role: FOLLOWER
I20260925 05:10:09.429390 22558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.430365 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.430773 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.431640 22558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:10:09.447685 22744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:09.447834 22744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:09.441654 22491 master_runner.cc:186] Error getting master registration for 127.8.215.61:42487: OK, instance_id { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" instance_seqno: 1790313008754598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:10:09.452654 22491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 05:10:09.485280 22512 tablet.cc:2369] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:09.486737 22632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } attempt: 5
Sep 25 05:10:09 dist-test-slave-wpdm krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
W20260925 05:10:09.503046 22660 tablet.cc:2369] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:09.505959 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.487570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.8.215.60:34189)
0925 05:10:09.487752 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.487759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.487781 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:10:09.487944 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.487949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.488037 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.488151 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.489407 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.490563 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.492361 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.493088 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.493228 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.497507 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.497517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.497524 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.497573 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.501386 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.502281 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.502287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.502291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.502440 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.502889 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.502894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.502898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.503359 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.503519 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.504899 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.505306 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:10:09.505643 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.487476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:34189 (local address 127.0.0.1:32930)
0925 05:10:09.487652 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.487673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:09.487810 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:09.488373 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:09.488379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:09.489007 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:09.489221 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.489234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.490732 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.490740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.492191 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.492208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.493250 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.493257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.493479 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.494177 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:09.494201 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:09.497249 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:09.501553 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.501560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.501565 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.502139 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.502563 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.502568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.502572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.502743 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.503599 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:09.503604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:09.503765 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 05:10:09.503926 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 05:10:09.504254 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 05:10:09.512560 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: Bootstrap starting.
I20260925 05:10:09.516842 22632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:09.518181 22632 log.cc:824] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:09.521626 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: No bootstrap required, opened a new log
I20260925 05:10:09.533017 22632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.533473 22632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:09.533604 22632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad14e1d486c431ea518da837108b1f2, State: Initialized, Role: FOLLOWER
I20260925 05:10:09.534327 22632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.535473 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.535923 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.536826 22632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:09.559509 22565 master_runner.cc:263] No leader master found from master 4ad14e1d486c431ea518da837108b1f2
I20260925 05:10:09.559746 22565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:10:09.560781 22719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:09.561094 22719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.562460 22719 leader_election.cc:288] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95db563521104233b68bef061ea351c8 (127.8.215.62:46521), 4ad14e1d486c431ea518da837108b1f2 (127.8.215.61:42487)
I20260925 05:10:09.563199 22533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f67c73b9e36480babbafdaf915580de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95db563521104233b68bef061ea351c8" is_pre_election: true
I20260925 05:10:09.563800 22533 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f67c73b9e36480babbafdaf915580de in term 0.
I20260925 05:10:09.564551 22655 leader_election.cc:302] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f67c73b9e36480babbafdaf915580de, 95db563521104233b68bef061ea351c8; no voters:
I20260925 05:10:09.565462 22607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f67c73b9e36480babbafdaf915580de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad14e1d486c431ea518da837108b1f2" is_pre_election: true
I20260925 05:10:09.566072 22607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f67c73b9e36480babbafdaf915580de in term 0.
W20260925 05:10:09.568626 22758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:09.568791 22758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:09.569959 22719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:09.570137 22719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:09.570282 22719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:09.572605 22719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.573815 22607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f67c73b9e36480babbafdaf915580de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad14e1d486c431ea518da837108b1f2"
I20260925 05:10:09.574055 22607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:09.576695 22533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f67c73b9e36480babbafdaf915580de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95db563521104233b68bef061ea351c8"
I20260925 05:10:09.576965 22533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:09.580531 22533 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f67c73b9e36480babbafdaf915580de in term 1.
I20260925 05:10:09.581079 22655 leader_election.cc:302] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f67c73b9e36480babbafdaf915580de, 95db563521104233b68bef061ea351c8; no voters:
I20260925 05:10:09.581485 22719 leader_election.cc:288] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [CANDIDATE]: Term 1 election: Requested vote from peers 95db563521104233b68bef061ea351c8 (127.8.215.62:46521), 4ad14e1d486c431ea518da837108b1f2 (127.8.215.61:42487)
I20260925 05:10:09.581699 22719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:09.581847 22719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [term 1 LEADER]: Becoming Leader. State: Replica: 2f67c73b9e36480babbafdaf915580de, State: Running, Role: LEADER
I20260925 05:10:09.582417 22719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } }
I20260925 05:10:09.584921 22607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f67c73b9e36480babbafdaf915580de in term 1.
I20260925 05:10:09.605477 22760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f67c73b9e36480babbafdaf915580de. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.605944 22760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:09.610384 22762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:09.612874 22762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:09.633512 22607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [term 1 FOLLOWER]: Refusing update from remote peer 2f67c73b9e36480babbafdaf915580de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:09.634542 22760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:09.641827 22533 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [term 1 FOLLOWER]: Refusing update from remote peer 2f67c73b9e36480babbafdaf915580de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:09.642711 22760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [LEADER]: Connected to new peer: Peer: permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:09.678839 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f67c73b9e36480babbafdaf915580de. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.679175 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.682698 22763 mvcc.cc:205] Tried to move back new op lower bound from 7333122087446237184 to 7333122087264751616. Current Snapshot: MvccSnapshot[applied={T|T < 7333122087446237184}]
I20260925 05:10:09.687259 22765 mvcc.cc:205] Tried to move back new op lower bound from 7333122087446237184 to 7333122087264751616. Current Snapshot: MvccSnapshot[applied={T|T < 7333122087446237184}]
I20260925 05:10:09.693992 22719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.694366 22719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:09.697448 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.697743 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.699594 22762 catalog_manager.cc:1347] Generated new cluster ID: c4a217e3d88940a5ba1266524212ca34
I20260925 05:10:09.699700 22762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:09.715508 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f67c73b9e36480babbafdaf915580de. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.715849 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.724676 22760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.725649 22760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:09.754073 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f67c73b9e36480babbafdaf915580de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db563521104233b68bef061ea351c8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 46521 } } peers { permanent_uuid: "4ad14e1d486c431ea518da837108b1f2" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 42487 } } peers { permanent_uuid: "2f67c73b9e36480babbafdaf915580de" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 34189 } } }
I20260925 05:10:09.754392 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:09.842020 22762 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:09.844064 22762 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:09.872578 22762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de: Generated new TSK 0
I20260925 05:10:09.873816 22762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:10.449819 22744 catalog_manager.cc:1259] Loaded cluster ID: c4a217e3d88940a5ba1266524212ca34
I20260925 05:10:10.449991 22744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: loading cluster ID for follower catalog manager: success
I20260925 05:10:10.452491 22744 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: acquiring CA information for follower catalog manager: success
I20260925 05:10:10.454299 22744 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95db563521104233b68bef061ea351c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:10:10.570753 22758 catalog_manager.cc:1259] Loaded cluster ID: c4a217e3d88940a5ba1266524212ca34
I20260925 05:10:10.570861 22758 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: loading cluster ID for follower catalog manager: success
I20260925 05:10:10.573220 22758 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: acquiring CA information for follower catalog manager: success
I20260925 05:10:10.574934 22758 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ad14e1d486c431ea518da837108b1f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:10:11.006310 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22491
I20260925 05:10:11.007536 22553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:11.118610 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22491
I20260925 05:10:11.141137 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22565
I20260925 05:10:11.142257 22627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:11.250720 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22565
I20260925 05:10:11.258195 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.257467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:46521 (local address 127.0.0.1:54326)
0925 05:10:11.257925 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:11.258037 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:46521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":256,"threads_started":1}
W20260925 05:10:11.258569 22655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.62:46521: connect: Connection refused (error 111)
I20260925 05:10:11.260066 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.259734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:46521 (local address 127.0.0.1:54330)
0925 05:10:11.259874 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:11.259955 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:46521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:10:11.260293 22655 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2f67c73b9e36480babbafdaf915580de -> Peer 95db563521104233b68bef061ea351c8 (127.8.215.62:46521): Couldn't send request to peer 95db563521104233b68bef061ea351c8. Status: Network error: Client connection negotiation failed: client connection to 127.8.215.62:46521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:10:11.274821 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22639
I20260925 05:10:11.275885 22701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:11.387811 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22639
2026-09-25T05:10:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4951 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:11 dist-test-slave-wpdm krb5kdc[22782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:11 dist-test-slave-wpdm krb5kdc[22782](info): set up 2 sockets
Sep 25 05:10:11 dist-test-slave-wpdm krb5kdc[22782](info): commencing operation
krb5kdc: starting...
W20260925 05:10:12.505913 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:10:12 dist-test-slave-wpdm krb5kdc[22782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:10:12.752889 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:36099
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:13.092967 22795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:13.093349 22795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:13.093549 22795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:13.100996 22795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:13.101171 22795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:13.101295 22795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:13.101402 22795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:13.101495 22795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:13.111958 22795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:13.113440 22795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:13.114880 22795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:13.127524 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.127554 22803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.127592 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:13.128554 22795 server_base.cc:1034] running on GCE node
I20260925 05:10:13.129559 22795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:13.131143 22795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:13.132424 22795 hybrid_clock.cc:648] HybridClock initialized: now 1790313013132417 us; error 113 us; skew 500 ppm
I20260925 05:10:13.133011 22795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:13 dist-test-slave-wpdm krb5kdc[22782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:13.141543 22795 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:13.143249 22795 webserver.cc:460] Webserver started at http://127.8.215.62:45447/ using document root <none> and password file <none>
I20260925 05:10:13.144019 22795 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:13.144204 22795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:13.144551 22795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:13.147966 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "338104c404e1475cbde51f0f641eb955"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-wpdm"
server_key: "b77bc39cebc1265e63bd081f0d55d30b"
server_key_iv: "ff83c41159a1bdc62f8eabdd5b2f3363"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.148854 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "338104c404e1475cbde51f0f641eb955"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-wpdm"
server_key: "b77bc39cebc1265e63bd081f0d55d30b"
server_key_iv: "ff83c41159a1bdc62f8eabdd5b2f3363"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.154374 22795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:10:13.158116 22810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:13.159669 22795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:10:13.159888 22795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "338104c404e1475cbde51f0f641eb955"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-wpdm"
server_key: "b77bc39cebc1265e63bd081f0d55d30b"
server_key_iv: "ff83c41159a1bdc62f8eabdd5b2f3363"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.160131 22795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:13.175765 22795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:13.180666 22795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:13.181010 22795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:13.202157 22795 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:36099
I20260925 05:10:13.202149 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:36099 every 8 connection(s)
I20260925 05:10:13.203752 22795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:13.205643 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22795
I20260925 05:10:13.205901 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:10:13.206616 9052 external_mini_cluster.cc:1428] Setting key 9d51e9b6c1eb0c7449972235277ff921
I20260925 05:10:13.208142 22862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:13.222193 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955: Bootstrap starting.
Sep 25 05:10:13 dist-test-slave-wpdm krb5kdc[22782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:13.226265 22862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:13.227663 22862 log.cc:824] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:13.231428 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955: No bootstrap required, opened a new log
I20260925 05:10:13.233204 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:13.208909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.8.215.62:36099)
0925 05:10:13.209736 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:13.209752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:13.209804 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:10:13.211673 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:13.211702 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:13.212253 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:13.212742 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:13.214330 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.216414 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.218027 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.219154 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.219355 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:13.223723 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:13.223738 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:13.223765 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:13.223826 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:13.227991 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.229132 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.229141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.229151 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.229423 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.229824 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.229829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.229832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.230547 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:13.230858 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:13.231259 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 05:10:13.231742 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":344,"threads_started":1}
I20260925 05:10:13.237493 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:13.240710 22862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338104c404e1475cbde51f0f641eb955" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36099 } }
I20260925 05:10:13.241168 22862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:13.241286 22862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338104c404e1475cbde51f0f641eb955, State: Initialized, Role: FOLLOWER
I20260925 05:10:13.241859 22862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338104c404e1475cbde51f0f641eb955" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36099 } }
I20260925 05:10:13.242064 22862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:13.242224 22862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:13.242399 22862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:13.244583 22862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338104c404e1475cbde51f0f641eb955" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36099 } }
I20260925 05:10:13.245185 22862 leader_election.cc:302] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 338104c404e1475cbde51f0f641eb955; no voters:
I20260925 05:10:13.245841 22862 leader_election.cc:288] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:13.246021 22867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:13.246847 22867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [term 1 LEADER]: Becoming Leader. State: Replica: 338104c404e1475cbde51f0f641eb955, State: Running, Role: LEADER
I20260925 05:10:13.247406 22867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338104c404e1475cbde51f0f641eb955" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36099 } }
I20260925 05:10:13.247974 22862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:13.251006 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 338104c404e1475cbde51f0f641eb955. Latest consensus state: current_term: 1 leader_uuid: "338104c404e1475cbde51f0f641eb955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338104c404e1475cbde51f0f641eb955" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36099 } } }
I20260925 05:10:13.250979 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "338104c404e1475cbde51f0f641eb955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338104c404e1475cbde51f0f641eb955" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36099 } } }
I20260925 05:10:13.251479 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:13.251493 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:13.253036 22874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:13.257696 22874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:13.267396 22874 catalog_manager.cc:1347] Generated new cluster ID: be4eb4ea26324c8796443bad2de0f009
I20260925 05:10:13.267551 22874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:13.280180 22874 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:13.282102 22874 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:13.291136 22874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 338104c404e1475cbde51f0f641eb955: Generated new TSK 0
I20260925 05:10:13.292291 22874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:13.349936 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22795
I20260925 05:10:13.351176 22857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:13.450290 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22795
2026-09-25T05:10:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2058 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:13 dist-test-slave-wpdm krb5kdc[22890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:13 dist-test-slave-wpdm krb5kdc[22890](info): set up 2 sockets
Sep 25 05:10:13 dist-test-slave-wpdm krb5kdc[22890](info): commencing operation
krb5kdc: starting...
W20260925 05:10:14.565821 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:10:14 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:10:14.820709 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43921
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:15.161610 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:15.161973 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:15.162127 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:15.169538 22903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:15.169704 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:15.169821 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:15.169919 22903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:15.169991 22903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:15.180240 22903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:15.181705 22903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:15.183095 22903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:15.195393 22908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:15.195348 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:15.195580 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:15.196106 22903 server_base.cc:1034] running on GCE node
I20260925 05:10:15.197044 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:15.198527 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:15.199769 22903 hybrid_clock.cc:648] HybridClock initialized: now 1790313015199707 us; error 54 us; skew 500 ppm
I20260925 05:10:15.200130 22903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:15 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:15.203333 22903 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:15.204931 22903 webserver.cc:460] Webserver started at http://127.8.215.62:37193/ using document root <none> and password file <none>
I20260925 05:10:15.205636 22903 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:15.205775 22903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:15.206084 22903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:15.209297 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "f060c1b53a6741edbe2ab7ae1257e60d"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "3a3765d1e61f940dec2217bf2c83e46d"
server_key_iv: "f046a4ab6943677e8c6e801044fe8e24"
server_key_version: "encryptionkey@0"
I20260925 05:10:15.210194 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "f060c1b53a6741edbe2ab7ae1257e60d"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "3a3765d1e61f940dec2217bf2c83e46d"
server_key_iv: "f046a4ab6943677e8c6e801044fe8e24"
server_key_version: "encryptionkey@0"
I20260925 05:10:15.215673 22903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:10:15.219754 22918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:15.221248 22903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:15.221442 22903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "f060c1b53a6741edbe2ab7ae1257e60d"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "3a3765d1e61f940dec2217bf2c83e46d"
server_key_iv: "f046a4ab6943677e8c6e801044fe8e24"
server_key_version: "encryptionkey@0"
I20260925 05:10:15.221664 22903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:15.235430 22903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:15.239681 22903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:15.239979 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:15.258827 22903 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43921
I20260925 05:10:15.258859 22969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43921 every 8 connection(s)
I20260925 05:10:15.260316 22903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:15.264197 22970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:15.264919 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22903
I20260925 05:10:15.265139 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:10:15.265803 9052 external_mini_cluster.cc:1428] Setting key 101d4ffbcc35be27c6083d9506a9ce47
I20260925 05:10:15.279207 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap starting.
Sep 25 05:10:15 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:15.282624 22970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:15.284152 22970 log.cc:824] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:15.287779 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: No bootstrap required, opened a new log
I20260925 05:10:15.291721 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:15.268018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.8.215.62:43921)
0925 05:10:15.268890 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:15.268906 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:15.268966 (+ 60us) server_negotiation.cc:407] Connection header received
0925 05:10:15.271096 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:15.271129 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:15.271706 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:15.272190 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:15.273634 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:15.275673 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:15.277102 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:15.278166 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:15.278364 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:15.282508 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:15.282520 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:15.282541 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:15.282594 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:15.286558 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:15.287357 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:15.287368 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:15.287377 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:15.287615 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:15.288145 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:15.288150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:15.288152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:15.288975 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:15.289307 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:15.289716 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 05:10:15.290221 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":344,"threads_started":1}
I20260925 05:10:15.297538 22970 raft_consensus.cc:348] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:15.298082 22970 raft_consensus.cc:374] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:15.298215 22970 raft_consensus.cc:729] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f060c1b53a6741edbe2ab7ae1257e60d, State: Initialized, Role: FOLLOWER
I20260925 05:10:15.298840 22970 consensus_queue.cc:260] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:15.299080 22970 raft_consensus.cc:388] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:15.299232 22970 raft_consensus.cc:482] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:15.299399 22970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:15.301710 22970 raft_consensus.cc:504] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:15.302237 22970 leader_election.cc:302] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f060c1b53a6741edbe2ab7ae1257e60d; no voters:
I20260925 05:10:15.302884 22970 leader_election.cc:288] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:15.303020 22975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:15.303797 22975 raft_consensus.cc:686] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 LEADER]: Becoming Leader. State: Replica: f060c1b53a6741edbe2ab7ae1257e60d, State: Running, Role: LEADER
I20260925 05:10:15.304509 22975 consensus_queue.cc:237] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:15.305294 22970 sys_catalog.cc:564] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:15.308079 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f060c1b53a6741edbe2ab7ae1257e60d. Latest consensus state: current_term: 1 leader_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } } }
I20260925 05:10:15.308254 22976 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } } }
I20260925 05:10:15.308518 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:15.308624 22976 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:15.318387 22983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:15.323020 22983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:15.332063 22983 catalog_manager.cc:1347] Generated new cluster ID: 382241bf3e114f0685b6f76ad7060645
I20260925 05:10:15.332212 22983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:15.346467 22983 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:15.348466 22983 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:15.357928 22983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Generated new TSK 0
I20260925 05:10:15.359047 22983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:10:15.428839 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43921
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:15.773870 22996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:15.774164 22996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:15.774276 22996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:15.781466 22996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:15.781598 22996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:15.781726 22996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:10:15.792232 22996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:15.793771 22996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:15.795066 22996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:15.808350 23002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:15.808458 23001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:15.808455 23004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:15.809463 22996 server_base.cc:1034] running on GCE node
I20260925 05:10:15.810204 22996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:15.811250 22996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:15.812495 22996 hybrid_clock.cc:648] HybridClock initialized: now 1790313015812455 us; error 46 us; skew 500 ppm
I20260925 05:10:15.812978 22996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:15 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:15.820475 22996 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:10:15.822252 22996 webserver.cc:460] Webserver started at http://127.8.215.1:40739/ using document root <none> and password file <none>
I20260925 05:10:15.823035 22996 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:15.823181 22996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:15.823494 22996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:15.826762 22996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "28e8ef0662d5451e8148c98612a3cdbb"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "f2b6d269f7994de3648d032cc18bee80"
server_key_iv: "58f313ac594633aca093e52c887cfed8"
server_key_version: "encryptionkey@0"
I20260925 05:10:15.827560 22996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "28e8ef0662d5451e8148c98612a3cdbb"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "f2b6d269f7994de3648d032cc18bee80"
server_key_iv: "58f313ac594633aca093e52c887cfed8"
server_key_version: "encryptionkey@0"
I20260925 05:10:15.833379 22996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:10:15.837877 23011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:15.840147 22996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:10:15.840344 22996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "28e8ef0662d5451e8148c98612a3cdbb"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "f2b6d269f7994de3648d032cc18bee80"
server_key_iv: "58f313ac594633aca093e52c887cfed8"
server_key_version: "encryptionkey@0"
I20260925 05:10:15.840853 22996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:15.896095 22996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:15.900514 22996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:15.900843 22996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:15.901821 22996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:15.903156 22996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:15.903230 22996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:15.903386 22996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:15.903455 22996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:15.937016 22996 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:46583
I20260925 05:10:15.937057 23123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:46583 every 8 connection(s)
I20260925 05:10:15.938329 22996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:15.945313 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 22996
I20260925 05:10:15.945555 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:15.946208 9052 external_mini_cluster.cc:1428] Setting key d89cf843ddb367c94ea72906eba1c4aa
Sep 25 05:10:15 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:15.959998 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:15.940921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:49953 (local address 127.8.215.62:43921)
0925 05:10:15.941414 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:15.941419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:15.942370 (+ 951us) server_negotiation.cc:407] Connection header received
0925 05:10:15.944259 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:15.944265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:15.944360 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:15.944477 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:15.946146 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:15.946968 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:15.949441 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:15.949851 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:15.949958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:15.954312 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:15.954317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:15.954320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:15.954353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:15.956930 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:15.957684 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:15.957689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:15.957691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:15.957809 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:15.958247 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:15.958251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:15.958254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:15.958510 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:15.958665 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:15.959384 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 05:10:15.959621 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260925 05:10:15.961174 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:15.941487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:49953)
0925 05:10:15.942185 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:15.942228 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:15.943973 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:15.944768 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:15.944786 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:15.945318 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:15.945974 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:15.945992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:15.947222 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:15.947230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:15.949282 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:15.949303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:15.950092 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:15.950102 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:15.950464 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:15.951236 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:15.951273 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:15.954047 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:15.957086 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:15.957096 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:15.957114 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:15.957548 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:15.957919 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:15.957924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:15.957928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:15.958119 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:15.958640 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:15.958648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:15.959002 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 05:10:15.959263 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 05:10:15.959832 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260925 05:10:15.962783 23124 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43921
I20260925 05:10:15.963140 23124 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:15.964044 23124 heartbeater.cc:511] Master 127.8.215.62:43921 requested a full tablet report, sending...
I20260925 05:10:15.966267 22935 ts_manager.cc:194] Registered new tserver with Master: 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583)
I20260925 05:10:15.967705 22935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:49953
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:10:16.004132 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43921
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:16.344306 23130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:16.344627 23130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:16.344813 23130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:16.352413 23130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:16.352541 23130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:16.352823 23130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:10:16.363057 23130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:16.364408 23130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:16.365832 23130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:16.379680 23135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.379716 23138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.379583 23136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.380846 23130 server_base.cc:1034] running on GCE node
I20260925 05:10:16.381444 23130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:16.382411 23130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:16.383608 23130 hybrid_clock.cc:648] HybridClock initialized: now 1790313016383543 us; error 64 us; skew 500 ppm
I20260925 05:10:16.383935 23130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:16 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:16.387146 23130 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:10:16.388830 23130 webserver.cc:460] Webserver started at http://127.8.215.2:34691/ using document root <none> and password file <none>
I20260925 05:10:16.389534 23130 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:16.389633 23130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:16.389887 23130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:16.392848 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "1c7bde983544438fb63ebba7e7db20c6"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "20134d3bfcc48253c04d589c5ca4919f"
server_key_iv: "1c1cbdc3772a8960a155b23786ae4bd9"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.393575 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "1c7bde983544438fb63ebba7e7db20c6"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "20134d3bfcc48253c04d589c5ca4919f"
server_key_iv: "1c1cbdc3772a8960a155b23786ae4bd9"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.398627 23130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 05:10:16.402048 23145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.403450 23130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:10:16.403592 23130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "1c7bde983544438fb63ebba7e7db20c6"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "20134d3bfcc48253c04d589c5ca4919f"
server_key_iv: "1c1cbdc3772a8960a155b23786ae4bd9"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.403769 23130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:16.431142 23130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:16.435384 23130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:16.435635 23130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:16.436558 23130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:16.437983 23130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:16.438056 23130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.438143 23130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:16.438187 23130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.473035 23130 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:36785
I20260925 05:10:16.473138 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:36785 every 8 connection(s)
I20260925 05:10:16.474400 23130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:16.480878 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23130
I20260925 05:10:16.481096 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:16.481755 9052 external_mini_cluster.cc:1428] Setting key 0a396711d6eea879ea6772b6768ebbb5
Sep 25 05:10:16 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:16.501360 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.477099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:45239 (local address 127.8.215.62:43921)
0925 05:10:16.477619 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.477624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.478488 (+ 864us) server_negotiation.cc:407] Connection header received
0925 05:10:16.480610 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.480615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.480783 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.480894 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.483002 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.484000 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.486916 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.487403 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.487513 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.493164 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.493176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.493185 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.493241 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.497619 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.498434 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.498444 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.498451 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.498662 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.499097 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.499106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.499113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.499573 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.499751 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.500477 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.500806 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
I20260925 05:10:16.501998 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.477526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:45239)
0925 05:10:16.478306 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.478358 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.480245 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.481235 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.481251 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.482077 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.482851 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.482870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.484179 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.484185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.486708 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.486729 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.487665 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.487676 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.488120 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.489061 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.489102 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.492835 (+ 3733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.497795 (+ 4960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.497803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.497818 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.498282 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.498784 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.498788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.498790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.498969 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.499698 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.499704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.500080 (+ 376us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.500328 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.500822 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":297,"threads_started":1}
I20260925 05:10:16.503599 23258 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43921
I20260925 05:10:16.503917 23258 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:16.504637 23258 heartbeater.cc:511] Master 127.8.215.62:43921 requested a full tablet report, sending...
I20260925 05:10:16.506291 22935 ts_manager.cc:194] Registered new tserver with Master: 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
I20260925 05:10:16.507225 22935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:45239
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:10:16.544152 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43921
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:16.879835 23264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:16.880124 23264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:16.880221 23264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:16.887598 23264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:16.887727 23264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:16.887845 23264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:10:16.898121 23264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:16.899420 23264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:16.900879 23264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:16.914819 23272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.914784 23269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.914875 23270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.918380 23264 server_base.cc:1034] running on GCE node
I20260925 05:10:16.919085 23264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:16.920113 23264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:16.921340 23264 hybrid_clock.cc:648] HybridClock initialized: now 1790313016921299 us; error 44 us; skew 500 ppm
I20260925 05:10:16.921696 23264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:16 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:16.925185 23264 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:10:16.926889 23264 webserver.cc:460] Webserver started at http://127.8.215.3:45983/ using document root <none> and password file <none>
I20260925 05:10:16.927680 23264 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:16.927825 23264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:16.928138 23264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:16.931267 23264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "ce4f8a2ca50349d488717a917afa86ba"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "37e0772b14b355d332dddd814a11a0aa"
server_key_iv: "bb9459727f4828b424ac200c8c2c6f4d"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.932054 23264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "ce4f8a2ca50349d488717a917afa86ba"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "37e0772b14b355d332dddd814a11a0aa"
server_key_iv: "bb9459727f4828b424ac200c8c2c6f4d"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.938998 23264 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.006s
I20260925 05:10:16.944145 23279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.945865 23264 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 05:10:16.946055 23264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "ce4f8a2ca50349d488717a917afa86ba"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "37e0772b14b355d332dddd814a11a0aa"
server_key_iv: "bb9459727f4828b424ac200c8c2c6f4d"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.946302 23264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:16.971443 23124 heartbeater.cc:503] Master 127.8.215.62:43921 was elected leader, sending a full tablet report...
I20260925 05:10:16.973716 23264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:16.978274 23264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:16.978541 23264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:16.979532 23264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:16.981045 23264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:16.981124 23264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.981220 23264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:16.981268 23264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:17.017994 23264 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:37747
I20260925 05:10:17.018030 23391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:37747 every 8 connection(s)
I20260925 05:10:17.019402 23264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:17.029400 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23264
I20260925 05:10:17.029584 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:17.030019 9052 external_mini_cluster.cc:1428] Setting key 1dca5d013e997ff918f7f7ab603b8a80
Sep 25 05:10:17 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:17.040381 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.022113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:35129 (local address 127.8.215.62:43921)
0925 05:10:17.022601 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.022606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.023755 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 05:10:17.025822 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.025828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.025941 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.026081 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:17.027722 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.028524 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.030394 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.030893 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.031035 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.034978 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:17.034983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:17.034987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:17.035023 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:17.037620 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.038327 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.038330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.038333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.038455 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.038854 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.038857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.038859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.039133 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:17.039254 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.039917 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.040089 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260925 05:10:17.041436 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.022755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:35129)
0925 05:10:17.023505 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.023604 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.025502 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.026332 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.026347 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.026920 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:17.027530 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.027548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.028687 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.028698 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.030248 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.030260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.031159 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.031166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.031510 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.032250 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:17.032285 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:17.034771 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:17.037761 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:17.037768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:17.037783 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:17.038207 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:17.038546 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:17.038551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:17.038554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:17.038757 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:17.039241 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:17.039246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:17.039575 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.039825 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.040231 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":295,"threads_started":1}
I20260925 05:10:17.042850 23392 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43921
I20260925 05:10:17.043174 23392 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:17.043926 23392 heartbeater.cc:511] Master 127.8.215.62:43921 requested a full tablet report, sending...
I20260925 05:10:17.045581 22935 ts_manager.cc:194] Registered new tserver with Master: ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747)
I20260925 05:10:17.046221 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:17.046597 22935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:35129
I20260925 05:10:17.064174 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.051294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.8.215.62:43921)
0925 05:10:17.051519 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.051529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.051983 (+ 454us) server_negotiation.cc:407] Connection header received
0925 05:10:17.052150 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.052154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.052219 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.052306 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:17.053769 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.054510 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.055800 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.056206 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.056307 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.058878 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:17.058882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:17.058886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:17.058919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:17.061504 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.062220 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.062224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.062226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.062361 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.062697 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.062701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.062702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.063044 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:17.063236 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.063567 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.063816 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:10:17.068490 22935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:17.075600 22935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:17.101111 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.088858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.8.215.1:46583)
0925 05:10:17.092121 (+ 3263us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.092131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.092168 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:10:17.092285 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.092294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.092622 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.093571 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:17.093815 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.094972 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.098338 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.099482 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.099668 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.099952 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.100479 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.100683 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3063,"thread_start_us":2892,"threads_started":1}
I20260925 05:10:17.101696 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.088507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45656)
0925 05:10:17.090078 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.090146 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.090392 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.092928 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.092937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.092980 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:17.093395 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.093405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.095144 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.095154 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.098166 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.098176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.099706 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.099713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.099954 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.100077 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.100279 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.101209 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":704,"thread_start_us":1178,"threads_started":1}
I20260925 05:10:17.104974 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.089721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.8.215.3:37747)
0925 05:10:17.096541 (+ 6820us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.096552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.096587 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:10:17.096713 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.096750 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.097015 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.098005 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:17.098310 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.099812 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.102418 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.103784 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.103938 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.104192 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.104437 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.104614 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6620,"thread_start_us":6428,"threads_started":1}
I20260925 05:10:17.105067 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.089236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37747 (local address 127.0.0.1:54280)
0925 05:10:17.091325 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.091343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.091497 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.097266 (+ 5769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.097272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.097301 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:17.097880 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.097893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.099974 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.099980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.102257 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.102270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.104016 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.104024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.104213 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.104255 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.104366 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.104520 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1956,"mutex_wait_us":6,"thread_start_us":1783,"threads_started":1}
I20260925 05:10:17.106154 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.090762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36785 (local address 127.0.0.1:50956)
0925 05:10:17.095724 (+ 4962us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.095746 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.095885 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.097221 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.097225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.097253 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:17.097725 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.097745 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.099786 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.099791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.103004 (+ 3213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.103021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.105064 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.105071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.105448 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.105480 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.105571 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.105718 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4791,"mutex_wait_us":117,"thread_start_us":4593,"threads_started":1}
I20260925 05:10:17.107267 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.091166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.8.215.2:36785)
0925 05:10:17.091864 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.091875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.096507 (+ 4632us) server_negotiation.cc:407] Connection header received
0925 05:10:17.096631 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.096639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.096995 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.097226 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:17.097925 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.099616 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.103191 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.104506 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.104641 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.104885 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.106714 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.106920 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":342,"threads_started":1}
I20260925 05:10:17.106729 23059 tablet_service.cc:1467] Processing CreateTablet for tablet a071578118a343489ec532e321f07791 (DEFAULT_TABLE table=test-table [id=b56c6f9a526947008135bb300726aa0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:17.109419 23059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a071578118a343489ec532e321f07791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:17.109849 23327 tablet_service.cc:1467] Processing CreateTablet for tablet a071578118a343489ec532e321f07791 (DEFAULT_TABLE table=test-table [id=b56c6f9a526947008135bb300726aa0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:17.111526 23327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a071578118a343489ec532e321f07791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:17.111342 23193 tablet_service.cc:1467] Processing CreateTablet for tablet a071578118a343489ec532e321f07791 (DEFAULT_TABLE table=test-table [id=b56c6f9a526947008135bb300726aa0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:17.113344 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a071578118a343489ec532e321f07791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:17.121955 23411 tablet_bootstrap.cc:492] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Bootstrap starting.
I20260925 05:10:17.125351 23411 tablet_bootstrap.cc:654] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:17.126593 23411 log.cc:824] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:17.127184 23412 tablet_bootstrap.cc:492] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Bootstrap starting.
I20260925 05:10:17.128644 23414 tablet_bootstrap.cc:492] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Bootstrap starting.
I20260925 05:10:17.131728 23414 tablet_bootstrap.cc:654] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:17.132977 23412 tablet_bootstrap.cc:654] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:17.133154 23414 log.cc:824] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:17.134197 23412 log.cc:824] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:17.135013 23411 tablet_bootstrap.cc:492] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: No bootstrap required, opened a new log
I20260925 05:10:17.135303 23411 ts_tablet_manager.cc:1397] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260925 05:10:17.135983 23414 tablet_bootstrap.cc:492] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: No bootstrap required, opened a new log
I20260925 05:10:17.136247 23414 ts_tablet_manager.cc:1397] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:10:17.137249 23412 tablet_bootstrap.cc:492] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: No bootstrap required, opened a new log
I20260925 05:10:17.137494 23412 ts_tablet_manager.cc:1397] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 05:10:17.143725 23414 raft_consensus.cc:348] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.144161 23414 raft_consensus.cc:374] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:17.144260 23414 raft_consensus.cc:729] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7bde983544438fb63ebba7e7db20c6, State: Initialized, Role: FOLLOWER
I20260925 05:10:17.144971 23414 consensus_queue.cc:260] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.146121 23258 heartbeater.cc:503] Master 127.8.215.62:43921 was elected leader, sending a full tablet report...
I20260925 05:10:17.146194 23411 raft_consensus.cc:348] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.146670 23411 raft_consensus.cc:374] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:17.146793 23411 raft_consensus.cc:729] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8ef0662d5451e8148c98612a3cdbb, State: Initialized, Role: FOLLOWER
I20260925 05:10:17.147609 23411 consensus_queue.cc:260] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.148053 23414 ts_tablet_manager.cc:1428] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 05:10:17.149277 23412 raft_consensus.cc:348] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.149770 23412 raft_consensus.cc:374] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:17.149865 23412 raft_consensus.cc:729] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4f8a2ca50349d488717a917afa86ba, State: Initialized, Role: FOLLOWER
I20260925 05:10:17.150034 23411 ts_tablet_manager.cc:1428] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260925 05:10:17.150615 23412 consensus_queue.cc:260] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.151820 23392 heartbeater.cc:503] Master 127.8.215.62:43921 was elected leader, sending a full tablet report...
I20260925 05:10:17.153816 23412 ts_tablet_manager.cc:1428] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260925 05:10:17.190833 23125 tablet.cc:2369] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:10:17.226483 23259 tablet.cc:2369] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:17.260475 23417 raft_consensus.cc:482] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:17.260860 23417 raft_consensus.cc:504] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.262926 23417 leader_election.cc:288] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583), ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747)
W20260925 05:10:17.271203 23393 tablet.cc:2369] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:17.271399 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.263238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37747 (local address 127.8.215.2:35555)
0925 05:10:17.264890 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.264908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.265038 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.265527 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.265530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.265548 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:17.265970 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.265980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.267438 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.267444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.269310 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.269320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.270473 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.270478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.270674 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.270699 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.270816 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.270954 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":514,"thread_start_us":311,"threads_started":1}
I20260925 05:10:17.271440 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.263547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:42391 (local address 127.8.215.1:46583)
0925 05:10:17.263737 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.263743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.264522 (+ 779us) server_negotiation.cc:407] Connection header received
0925 05:10:17.264891 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.264899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.264994 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.265099 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:17.265666 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.266605 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.268981 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.270226 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.270365 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.270471 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.271032 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.271189 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:10:17.272104 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.263637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:35555 (local address 127.8.215.3:37747)
0925 05:10:17.263898 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.263905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.265010 (+ 1105us) server_negotiation.cc:407] Connection header received
0925 05:10:17.265219 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.265224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.265306 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.265357 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:17.266153 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.267286 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.269458 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.270242 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.270397 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.270522 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.270840 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.271848 (+ 1008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 05:10:17.272506 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.263238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.8.215.2:42391)
0925 05:10:17.264402 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.264422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.264646 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.265114 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.265119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.265145 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:17.265501 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.265514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.266744 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.266751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.268799 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.268816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.270626 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.270632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.270848 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.270880 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.271991 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.272138 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":1,"thread_start_us":1264,"threads_started":1}
I20260925 05:10:17.273079 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071578118a343489ec532e321f07791" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4f8a2ca50349d488717a917afa86ba" is_pre_election: true
I20260925 05:10:17.273696 23347 raft_consensus.cc:2454] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 0.
I20260925 05:10:17.273656 23079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071578118a343489ec532e321f07791" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e8ef0662d5451e8148c98612a3cdbb" is_pre_election: true
I20260925 05:10:17.274277 23079 raft_consensus.cc:2454] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 0.
I20260925 05:10:17.274477 23148 leader_election.cc:302] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba; no voters:
I20260925 05:10:17.274900 23417 raft_consensus.cc:2781] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:17.275075 23417 raft_consensus.cc:482] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:17.275213 23417 raft_consensus.cc:3037] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:17.277519 23417 raft_consensus.cc:504] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.278182 23417 leader_election.cc:288] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 election: Requested vote from peers 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583), ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747)
I20260925 05:10:17.278658 23079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071578118a343489ec532e321f07791" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e8ef0662d5451e8148c98612a3cdbb"
I20260925 05:10:17.278717 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071578118a343489ec532e321f07791" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4f8a2ca50349d488717a917afa86ba"
I20260925 05:10:17.278862 23079 raft_consensus.cc:3037] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:17.278999 23347 raft_consensus.cc:3037] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:17.281204 23079 raft_consensus.cc:2454] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 1.
I20260925 05:10:17.281291 23347 raft_consensus.cc:2454] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 1.
I20260925 05:10:17.281742 23146 leader_election.cc:302] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7bde983544438fb63ebba7e7db20c6, 28e8ef0662d5451e8148c98612a3cdbb; no voters:
I20260925 05:10:17.282195 23417 raft_consensus.cc:2781] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:17.282765 23417 raft_consensus.cc:686] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7bde983544438fb63ebba7e7db20c6, State: Running, Role: LEADER
I20260925 05:10:17.283259 23417 consensus_queue.cc:237] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } }
I20260925 05:10:17.288384 22935 catalog_manager.cc:5495] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "1c7bde983544438fb63ebba7e7db20c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:17.340018 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.335182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.8.215.2:36785)
0925 05:10:17.335358 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.335362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.335379 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:10:17.335472 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.335475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.335554 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.335703 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:17.336019 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.336747 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.338136 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.338530 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.338626 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.339040 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:17.339557 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:17.339685 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.339726 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.339794 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:10:17.348910 23347 raft_consensus.cc:1261] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Refusing update from remote peer 1c7bde983544438fb63ebba7e7db20c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:17.348912 23079 raft_consensus.cc:1261] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Refusing update from remote peer 1c7bde983544438fb63ebba7e7db20c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:17.349960 23422 consensus_queue.cc:1035] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:17.350368 23417 consensus_queue.cc:1035] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:17.361063 23425 mvcc.cc:205] Tried to move back new op lower bound from 7333122119049506816 to 7333122118795816960. Current Snapshot: MvccSnapshot[applied={T|T < 7333122119049506816}]
I20260925 05:10:17.361810 23427 mvcc.cc:205] Tried to move back new op lower bound from 7333122119049506816 to 7333122118795816960. Current Snapshot: MvccSnapshot[applied={T|T < 7333122119049506816}]
I20260925 05:10:17.366648 23426 mvcc.cc:205] Tried to move back new op lower bound from 7333122119049506816 to 7333122118795816960. Current Snapshot: MvccSnapshot[applied={T|T < 7333122119049506816}]
I20260925 05:10:17.379565 22935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:17.379824 22935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:17.383582 22935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Sending DeleteTablet for 3 replicas of tablet a071578118a343489ec532e321f07791
I20260925 05:10:17.384868 23327 tablet_service.cc:1514] Processing DeleteTablet for tablet a071578118a343489ec532e321f07791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:17 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:54280
I20260925 05:10:17.384991 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22996
I20260925 05:10:17.384923 23059 tablet_service.cc:1514] Processing DeleteTablet for tablet a071578118a343489ec532e321f07791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:17 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:45656
I20260925 05:10:17.385921 23193 tablet_service.cc:1514] Processing DeleteTablet for tablet a071578118a343489ec532e321f07791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:17 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:50956
I20260925 05:10:17.397993 23435 tablet_replica.cc:330] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: stopping tablet replica
I20260925 05:10:17.398762 23436 tablet_replica.cc:330] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: stopping tablet replica
I20260925 05:10:17.399435 23438 tablet_replica.cc:330] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: stopping tablet replica
I20260925 05:10:17.399745 23435 raft_consensus.cc:2229] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:17.400353 23435 raft_consensus.cc:2258] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:17.400311 23436 raft_consensus.cc:2229] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:17.400619 23438 raft_consensus.cc:2229] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:17.401405 23438 raft_consensus.cc:2258] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:17.401989 23436 raft_consensus.cc:2258] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:17.403308 23435 ts_tablet_manager.cc:1905] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:17.404783 23438 ts_tablet_manager.cc:1905] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:17.406183 23436 ts_tablet_manager.cc:1905] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:17.410768 23435 ts_tablet_manager.cc:1918] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:17.410990 23435 log.cc:1196] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/a071578118a343489ec532e321f07791
I20260925 05:10:17.411594 23435 ts_tablet_manager.cc:1939] T a071578118a343489ec532e321f07791 P ce4f8a2ca50349d488717a917afa86ba: Deleting consensus metadata
Sep 25 05:10:17 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:10:17.413817 22921 catalog_manager.cc:4841] TS ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): tablet a071578118a343489ec532e321f07791 (table test-table [id=b56c6f9a526947008135bb300726aa0d]) successfully deleted
I20260925 05:10:17.415215 23438 ts_tablet_manager.cc:1918] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:17.415453 23438 log.cc:1196] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/a071578118a343489ec532e321f07791
I20260925 05:10:17.416061 23438 ts_tablet_manager.cc:1939] T a071578118a343489ec532e321f07791 P 1c7bde983544438fb63ebba7e7db20c6: Deleting consensus metadata
I20260925 05:10:17.418201 22919 catalog_manager.cc:4841] TS 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): tablet a071578118a343489ec532e321f07791 (table test-table [id=b56c6f9a526947008135bb300726aa0d]) successfully deleted
I20260925 05:10:17.418423 23436 ts_tablet_manager.cc:1918] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:17.418641 23436 log.cc:1196] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/a071578118a343489ec532e321f07791
I20260925 05:10:17.419124 23436 ts_tablet_manager.cc:1939] T a071578118a343489ec532e321f07791 P 28e8ef0662d5451e8148c98612a3cdbb: Deleting consensus metadata
I20260925 05:10:17.420892 22919 catalog_manager.cc:4841] TS 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583): tablet a071578118a343489ec532e321f07791 (table test-table [id=b56c6f9a526947008135bb300726aa0d]) successfully deleted
I20260925 05:10:17.423012 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.386286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.8.215.1:46583)
0925 05:10:17.386518 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.386527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.398280 (+ 11753us) server_negotiation.cc:407] Connection header received
0925 05:10:17.398596 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.398603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.398754 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.398843 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:17.400970 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.402139 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.405218 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.405951 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.406055 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.415163 (+ 9108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:17.415179 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:17.415200 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:17.415261 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:17.419671 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.420676 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.420687 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.420690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.420938 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.421431 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.421436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.421438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.421842 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:17.422081 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.422468 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.422682 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:10:17.423640 23119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:17.576915 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22996
I20260925 05:10:17.596133 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 23130
Sep 25 05:10:17 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:10:17.613412 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.597074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.8.215.2:36785)
0925 05:10:17.597271 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.597277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.597298 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:10:17.597458 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.597463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.597542 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.597653 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:17.598892 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.599710 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.601002 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.601478 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.601613 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.606963 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:17.606982 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:17.607002 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:17.607043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:17.610469 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.611181 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.611188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.611191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.611387 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.611778 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.611783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.611787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.612158 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:17.612504 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.612797 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.613022 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:10:17.614110 23253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:17.728609 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 23130
I20260925 05:10:17.747469 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 23264
Sep 25 05:10:17 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:10:17.761895 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.748276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.8.215.3:37747)
0925 05:10:17.748451 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.748457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.748505 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:10:17.748785 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.748795 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.748865 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.748949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:17.750246 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.750996 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.752237 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.752615 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.752713 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.757049 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:17.757063 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:17.757078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:17.757108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:17.759511 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.760115 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.760122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.760125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.760258 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:17.760562 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:17.760565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:17.760566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:17.761002 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:17.761211 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.761408 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.761557 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:10:17.762590 23387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:17.853235 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 23264
I20260925 05:10:17.869987 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 22903
I20260925 05:10:17.871057 22965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:17.980918 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 22903
I20260925 05:10:18.003147 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43921
--webserver_interface=127.8.215.62
--webserver_port=37193
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:18.335827 23443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:18.336112 23443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:18.336200 23443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:18.343618 23443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:18.343735 23443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:18.343784 23443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:18.343843 23443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:18.343888 23443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:18.354935 23443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=37193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:18.356356 23443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:18.357774 23443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:18.383322 23449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:18.383239 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:18.383255 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:18.384222 23443 server_base.cc:1034] running on GCE node
I20260925 05:10:18.385241 23443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:18.386909 23443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:18.388172 23443 hybrid_clock.cc:648] HybridClock initialized: now 1790313018388131 us; error 78 us; skew 500 ppm
I20260925 05:10:18.388618 23443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:18 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313018, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:18.397141 23443 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:18.398891 23443 webserver.cc:460] Webserver started at http://127.8.215.62:37193/ using document root <none> and password file <none>
I20260925 05:10:18.399718 23443 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:18.399986 23443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:18.406462 23443 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 05:10:18.409513 23459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.410951 23443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:18.411150 23443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "f060c1b53a6741edbe2ab7ae1257e60d"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "3a3765d1e61f940dec2217bf2c83e46d"
server_key_iv: "f046a4ab6943677e8c6e801044fe8e24"
server_key_version: "encryptionkey@0"
I20260925 05:10:18.412250 23443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:18.436688 23443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:18.441233 23443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:18.441529 23443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:18.462697 23443 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43921
I20260925 05:10:18.462716 23510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43921 every 8 connection(s)
I20260925 05:10:18.464257 23443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:18.467857 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23443
I20260925 05:10:18.468933 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:46583
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:43921
--webserver_port=40739
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:18.473862 23511 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:10:18.479085 23511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap starting.
I20260925 05:10:18.500034 23511 log.cc:824] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:18.515702 23511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:18.516407 23511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap complete.
I20260925 05:10:18.531275 23511 raft_consensus.cc:348] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:18.533017 23511 raft_consensus.cc:729] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f060c1b53a6741edbe2ab7ae1257e60d, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.533988 23511 consensus_queue.cc:260] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:18.534343 23511 raft_consensus.cc:388] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:18.534477 23511 raft_consensus.cc:482] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:18.534682 23511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:10:18.540474 23511 raft_consensus.cc:504] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:18.541185 23511 leader_election.cc:302] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f060c1b53a6741edbe2ab7ae1257e60d; no voters:
I20260925 05:10:18.541847 23511 leader_election.cc:288] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:10:18.542044 23515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:10:18.543354 23515 raft_consensus.cc:686] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 LEADER]: Becoming Leader. State: Replica: f060c1b53a6741edbe2ab7ae1257e60d, State: Running, Role: LEADER
I20260925 05:10:18.544113 23511 sys_catalog.cc:564] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:18.544008 23515 consensus_queue.cc:237] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:18.546627 23517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f060c1b53a6741edbe2ab7ae1257e60d. Latest consensus state: current_term: 2 leader_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } } }
I20260925 05:10:18.547062 23517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:18.547631 23516 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } } }
I20260925 05:10:18.547971 23516 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:18.557832 23522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:18.567075 23522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:18.568605 23522 catalog_manager.cc:1259] Loaded cluster ID: 382241bf3e114f0685b6f76ad7060645
I20260925 05:10:18.568713 23522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:18.572072 23522 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 05:10:18.590972 23522 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 05:10:18.610289 23522 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:18.612161 23522 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Loaded TSK: 0
I20260925 05:10:18.613737 23522 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:10:18.847296 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:18.847645 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:18.847843 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:18.855482 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:18.855646 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:18.855818 23513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:10:18.866357 23513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:46583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=40739
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:18.867777 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:18.869225 23513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:18.885080 23538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:18.885061 23541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:18.885154 23539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:18.886965 23513 server_base.cc:1034] running on GCE node
I20260925 05:10:18.887687 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:18.888775 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:18.890036 23513 hybrid_clock.cc:648] HybridClock initialized: now 1790313018889968 us; error 62 us; skew 500 ppm
I20260925 05:10:18.890376 23513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:18 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313018, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:18.893651 23513 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:10:18.895445 23513 webserver.cc:460] Webserver started at http://127.8.215.1:40739/ using document root <none> and password file <none>
I20260925 05:10:18.896198 23513 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:18.896345 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:18.902133 23513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:10:18.905157 23548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.906701 23513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:18.906901 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "28e8ef0662d5451e8148c98612a3cdbb"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "f2b6d269f7994de3648d032cc18bee80"
server_key_iv: "58f313ac594633aca093e52c887cfed8"
server_key_version: "encryptionkey@0"
I20260925 05:10:18.907884 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:18.929056 23513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:18.933732 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:18.934029 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:18.935325 23513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:18.937462 23513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:18.937599 23513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.937722 23513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:18.937803 23513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.978427 23513 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:46583
I20260925 05:10:18.978518 23660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:46583 every 8 connection(s)
I20260925 05:10:18.979954 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:18.987044 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23513
I20260925 05:10:18.987722 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:36785
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:43921
--webserver_port=34691
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:18 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313018, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:19.009615 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.983211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:35893)
0925 05:10:18.984152 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:18.984198 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:18.986899 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:18.989601 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:18.989609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:18.990134 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:18.990683 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.990702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.992700 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.992708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.994987 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.995008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.996145 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.996155 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.996494 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.997359 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:18.997383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:18.999639 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:19.005120 (+ 5481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:19.005128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:19.005142 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:19.005606 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:19.006200 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:19.006204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:19.006207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:19.006419 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:19.007508 (+ 1089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:19.007515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:19.007827 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 05:10:19.008056 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 05:10:19.008481 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":403,"threads_started":1}
I20260925 05:10:19.010386 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.982938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:35893 (local address 127.8.215.62:43921)
0925 05:10:18.983579 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.983591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.984335 (+ 744us) server_negotiation.cc:407] Connection header received
0925 05:10:18.987699 (+ 3364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.987722 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.988364 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.988889 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:18.990929 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.992470 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.995187 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.995885 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.996044 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.999971 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:18.999985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:19.000010 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:19.000075 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:19.004941 (+ 4866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.005782 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.005796 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.005809 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.006044 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.006578 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.006587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.006593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.007351 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:19.007587 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:19.008364 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 05:10:19.008934 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":299,"threads_started":1}
I20260925 05:10:19.013713 23661 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43921
I20260925 05:10:19.014127 23661 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:19.015225 23661 heartbeater.cc:511] Master 127.8.215.62:43921 requested a full tablet report, sending...
I20260925 05:10:19.018018 23476 ts_manager.cc:194] Registered new tserver with Master: 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583)
I20260925 05:10:19.019549 23476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:35893
W20260925 05:10:19.339290 23665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:19.339748 23665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:19.339943 23665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:19.347880 23665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:19.348035 23665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:19.348227 23665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:10:19.358525 23665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:36785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=34691
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:19.359900 23665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:19.361312 23665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:19.374727 23673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:19.374692 23670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:19.374809 23671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:19.375731 23665 server_base.cc:1034] running on GCE node
I20260925 05:10:19.376353 23665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:19.377350 23665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:19.378551 23665 hybrid_clock.cc:648] HybridClock initialized: now 1790313019378497 us; error 51 us; skew 500 ppm
I20260925 05:10:19.378873 23665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:19 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313019, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:19.382274 23665 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:10:19.383616 23665 webserver.cc:460] Webserver started at http://127.8.215.2:34691/ using document root <none> and password file <none>
I20260925 05:10:19.384275 23665 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:19.384384 23665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:19.389729 23665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 05:10:19.392323 23680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:19.393869 23665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:10:19.394006 23665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "1c7bde983544438fb63ebba7e7db20c6"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "20134d3bfcc48253c04d589c5ca4919f"
server_key_iv: "1c1cbdc3772a8960a155b23786ae4bd9"
server_key_version: "encryptionkey@0"
I20260925 05:10:19.394992 23665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:19.418485 23665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:19.422787 23665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:19.423012 23665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:19.423954 23665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:19.425421 23665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:19.425511 23665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:19.425592 23665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:19.425648 23665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:19.458575 23665 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:36785
I20260925 05:10:19.458662 23792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:36785 every 8 connection(s)
I20260925 05:10:19.460099 23665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:19.469378 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23665
I20260925 05:10:19.470037 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:37747
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:43921
--webserver_port=45983
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:19 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313019, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:19.484551 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.462815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:41081 (local address 127.8.215.62:43921)
0925 05:10:19.462986 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.462991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.464357 (+ 1366us) server_negotiation.cc:407] Connection header received
0925 05:10:19.466699 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.466705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.466814 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.466927 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:19.468471 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.469492 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.471793 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.472577 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.472774 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.477420 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:19.477427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:19.477433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:19.477474 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:19.481024 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.482074 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.482084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.482086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.482231 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.482718 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.482721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.482724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.482981 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:19.483090 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:19.484109 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 05:10:19.484301 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:10:19.486269 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.463429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:41081)
0925 05:10:19.464187 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:19.464231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:19.466409 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:19.467189 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:19.467198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:19.467718 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:19.468298 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.468315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.469643 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.469650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.471609 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.471626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.472834 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.472841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.473125 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.473940 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:19.473970 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:19.477182 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:19.481182 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:19.481194 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:19.481216 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:19.481940 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:19.482349 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:19.482356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:19.482363 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:19.482609 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:19.483115 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:19.483126 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:19.483607 (+ 481us) client_negotiation.cc:769] Sending connection context
0925 05:10:19.483963 (+ 356us) client_negotiation.cc:240] Negotiation successful
0925 05:10:19.484559 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":302,"threads_started":1}
I20260925 05:10:19.487815 23793 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43921
I20260925 05:10:19.488140 23793 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:19.489087 23793 heartbeater.cc:511] Master 127.8.215.62:43921 requested a full tablet report, sending...
I20260925 05:10:19.490767 23476 ts_manager.cc:194] Registered new tserver with Master: 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
I20260925 05:10:19.491602 23476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:41081
W20260925 05:10:19.811806 23796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:19.812142 23796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:19.812314 23796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:19.820097 23796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:19.820264 23796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:19.820439 23796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:10:19.830786 23796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:37747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=45983
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:19.832158 23796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:19.833619 23796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:19.847106 23801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:19.847215 23804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:19.847193 23802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:19.848037 23796 server_base.cc:1034] running on GCE node
I20260925 05:10:19.848845 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:19.849885 23796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:19.851116 23796 hybrid_clock.cc:648] HybridClock initialized: now 1790313019851074 us; error 32 us; skew 500 ppm
I20260925 05:10:19.851487 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:19 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313019, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:19.854754 23796 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:10:19.856336 23796 webserver.cc:460] Webserver started at http://127.8.215.3:45983/ using document root <none> and password file <none>
I20260925 05:10:19.857162 23796 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:19.857321 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:19.862957 23796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 05:10:19.865875 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:19.867257 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:10:19.867443 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "ce4f8a2ca50349d488717a917afa86ba"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "37e0772b14b355d332dddd814a11a0aa"
server_key_iv: "bb9459727f4828b424ac200c8c2c6f4d"
server_key_version: "encryptionkey@0"
I20260925 05:10:19.868543 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:19.881390 23796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:19.885281 23796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:19.885546 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:19.886512 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:19.887884 23796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:19.888005 23796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:19.888146 23796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:19.888234 23796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:19.921253 23796 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:37747
I20260925 05:10:19.921342 23923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:37747 every 8 connection(s)
I20260925 05:10:19.922924 23796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:19.929178 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23796
Sep 25 05:10:19 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313019, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:19.944675 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.930158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.8.215.62:43921)
0925 05:10:19.931143 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.931147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.931161 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:10:19.931203 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.931206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.931260 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.931392 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:19.932920 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.933642 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.935147 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.935821 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.935948 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.938818 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:19.938823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:19.938826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:19.938865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:19.941884 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.942570 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.942575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.942579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.942709 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.943091 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.943096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.943101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.943391 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:19.943644 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:19.943961 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 05:10:19.944206 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":893,"thread_start_us":788,"threads_started":1}
I20260925 05:10:19.944777 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.925596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:40715 (local address 127.8.215.62:43921)
0925 05:10:19.925817 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.925823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.927005 (+ 1182us) server_negotiation.cc:407] Connection header received
0925 05:10:19.929018 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.929025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.929123 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.929248 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:19.931274 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.932047 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.933720 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.934135 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.934287 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.938276 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:19.938281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:19.938284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:19.938313 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:19.941651 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.942402 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.942411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.942414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.942532 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:19.942971 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:19.942976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:19.942980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:19.943252 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:19.943412 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:19.944098 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 05:10:19.944337 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":72}
I20260925 05:10:19.946069 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.926105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:40715)
0925 05:10:19.926837 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:19.926883 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:19.928629 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:19.929505 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:19.929515 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:19.930146 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:19.931095 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.931122 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.932222 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.932228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.933560 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.933572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.934381 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.934389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.934661 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.935436 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:19.935468 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:19.938044 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:19.941795 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:19.941804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:19.941823 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:19.942270 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:19.942643 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:19.942651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:19.942655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:19.942850 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:19.943392 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:19.943400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:19.943738 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 05:10:19.944003 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 05:10:19.944641 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":279,"threads_started":1}
I20260925 05:10:19.947619 23924 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43921
I20260925 05:10:19.947937 23924 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:19.948694 23924 heartbeater.cc:511] Master 127.8.215.62:43921 requested a full tablet report, sending...
I20260925 05:10:19.950168 23476 ts_manager.cc:194] Registered new tserver with Master: ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747)
I20260925 05:10:19.950978 23476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:40715
I20260925 05:10:19.960021 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:19.966746 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.960962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.8.215.62:43921)
0925 05:10:19.961160 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.961169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.961602 (+ 433us) server_negotiation.cc:407] Connection header received
0925 05:10:19.961803 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.961807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.961942 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.962082 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:19.962500 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.963287 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.964680 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.965182 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.965289 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.965779 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:19.966288 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:19.966396 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:19.966489 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 05:10:19.966563 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:10:19.967648 23476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:19.975075 23476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:20.000522 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.987481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36785 (local address 127.0.0.1:50986)
0925 05:10:19.990801 (+ 3320us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:19.990825 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:19.990963 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:19.992850 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:19.992860 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:19.992896 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:19.993376 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.993388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.995188 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.995196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.997765 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.997779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.999536 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.999542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.999723 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.999775 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 05:10:19.999920 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 05:10:20.000130 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3160,"thread_start_us":1064,"threads_started":1}
I20260925 05:10:20.000869 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.988067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.8.215.2:36785)
0925 05:10:19.991798 (+ 3731us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.991810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.991843 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:10:19.991948 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.991956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.992252 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.992475 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:19.993582 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.995048 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.997936 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.999326 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.999476 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:19.999703 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:19.999978 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.000139 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":3429,"threads_started":1}
I20260925 05:10:20.001811 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.987402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37747 (local address 127.0.0.1:54302)
0925 05:10:19.989057 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:19.989113 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:19.989341 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:19.993886 (+ 4545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:19.993893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:19.993930 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:19.994376 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.994392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.996290 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.996296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:19.998729 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.998745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.000971 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.000980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.001155 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.001199 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 05:10:20.001294 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:10:20.001426 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":298,"thread_start_us":3068,"threads_started":1}
I20260925 05:10:20.001840 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.987608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.8.215.3:37747)
0925 05:10:19.993261 (+ 5653us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.993270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.993306 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:10:19.993400 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.993408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.993674 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.993863 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:19.994593 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.996116 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.998888 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.000395 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.000523 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.000808 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.001364 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.001539 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5474,"thread_start_us":5308,"threads_started":1}
I20260925 05:10:20.003929 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.992289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.8.215.1:46583)
0925 05:10:19.995603 (+ 3314us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:19.995611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:19.995637 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:10:19.995720 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:19.995727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:19.995925 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:19.996120 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:19.996953 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.998274 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.001124 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.002878 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.003013 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.003252 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.003476 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.003633 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3150,"thread_start_us":2984,"threads_started":1}
I20260925 05:10:20.003945 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:19.991596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45676)
0925 05:10:19.992080 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:19.992098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:19.992198 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:19.996130 (+ 3932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:19.996135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:19.996159 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:19.996668 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:19.996682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:19.998401 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:19.998409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.000511 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:20.000524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.003113 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.003121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.003295 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.003327 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:10:20.003423 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:10:20.003577 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":4,"thread_start_us":926,"threads_started":1}
I20260925 05:10:20.005045 23728 tablet_service.cc:1467] Processing CreateTablet for tablet 16c979bc982d45188024e512266e28f8 (DEFAULT_TABLE table=test-table [id=9a63169fbba14943a9faccaeb3aedd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:20.006242 23728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c979bc982d45188024e512266e28f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:20.005685 23859 tablet_service.cc:1467] Processing CreateTablet for tablet 16c979bc982d45188024e512266e28f8 (DEFAULT_TABLE table=test-table [id=9a63169fbba14943a9faccaeb3aedd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:20.007081 23859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c979bc982d45188024e512266e28f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:20.007681 23596 tablet_service.cc:1467] Processing CreateTablet for tablet 16c979bc982d45188024e512266e28f8 (DEFAULT_TABLE table=test-table [id=9a63169fbba14943a9faccaeb3aedd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:20.009089 23596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c979bc982d45188024e512266e28f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:20.018916 23940 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Bootstrap starting.
I20260925 05:10:20.022651 23941 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Bootstrap starting.
I20260925 05:10:20.023000 23940 tablet_bootstrap.cc:654] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:20.023660 23661 heartbeater.cc:503] Master 127.8.215.62:43921 was elected leader, sending a full tablet report...
I20260925 05:10:20.024812 23940 log.cc:824] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:20.025229 23942 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Bootstrap starting.
I20260925 05:10:20.027648 23941 tablet_bootstrap.cc:654] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:20.027961 23942 tablet_bootstrap.cc:654] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:20.029066 23942 log.cc:824] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:20.029204 23941 log.cc:824] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:20.032223 23940 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: No bootstrap required, opened a new log
I20260925 05:10:20.032614 23940 ts_tablet_manager.cc:1397] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260925 05:10:20.034895 23941 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: No bootstrap required, opened a new log
I20260925 05:10:20.035179 23941 ts_tablet_manager.cc:1397] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260925 05:10:20.035848 23942 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: No bootstrap required, opened a new log
I20260925 05:10:20.036108 23942 ts_tablet_manager.cc:1397] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260925 05:10:20.043423 23941 raft_consensus.cc:348] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.043958 23941 raft_consensus.cc:374] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:20.044065 23941 raft_consensus.cc:729] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4f8a2ca50349d488717a917afa86ba, State: Initialized, Role: FOLLOWER
I20260925 05:10:20.044124 23940 raft_consensus.cc:348] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.044570 23940 raft_consensus.cc:374] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:20.044675 23940 raft_consensus.cc:729] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7bde983544438fb63ebba7e7db20c6, State: Initialized, Role: FOLLOWER
I20260925 05:10:20.044764 23941 consensus_queue.cc:260] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.045176 23942 raft_consensus.cc:348] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.045367 23940 consensus_queue.cc:260] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.045611 23942 raft_consensus.cc:374] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:20.045719 23942 raft_consensus.cc:729] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8ef0662d5451e8148c98612a3cdbb, State: Initialized, Role: FOLLOWER
I20260925 05:10:20.046180 23924 heartbeater.cc:503] Master 127.8.215.62:43921 was elected leader, sending a full tablet report...
I20260925 05:10:20.046298 23942 consensus_queue.cc:260] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.046871 23941 ts_tablet_manager.cc:1428] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260925 05:10:20.047901 23793 heartbeater.cc:503] Master 127.8.215.62:43921 was elected leader, sending a full tablet report...
I20260925 05:10:20.048595 23940 ts_tablet_manager.cc:1428] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260925 05:10:20.050313 23942 ts_tablet_manager.cc:1428] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
W20260925 05:10:20.174571 23925 tablet.cc:2369] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:10:20.212188 23794 tablet.cc:2369] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:10:20.232363 23662 tablet.cc:2369] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:20.283275 23947 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:20.283604 23947 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.285454 23947 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583)
I20260925 05:10:20.292055 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.286220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:41419 (local address 127.8.215.1:46583)
0925 05:10:20.286531 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:20.286541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:20.286839 (+ 298us) server_negotiation.cc:407] Connection header received
0925 05:10:20.287075 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:20.287079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:20.287140 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:20.287242 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:20.287776 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.288510 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.290331 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.291100 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.291275 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.291390 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.291664 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.291801 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 05:10:20.292143 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.285838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.8.215.2:41419)
0925 05:10:20.286717 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.286742 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:20.286918 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:20.287261 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:20.287265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:20.287287 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:20.287626 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:20.287633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.288641 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.288649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.290165 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:20.290177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.291348 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.291354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.291501 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.291524 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:10:20.291627 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:10:20.291749 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":3,"thread_start_us":958,"threads_started":1}
I20260925 05:10:20.293244 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.285868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:56683 (local address 127.8.215.3:37747)
0925 05:10:20.286079 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:20.286087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:20.287274 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 05:10:20.287522 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:20.287527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:20.287597 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:20.287719 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:20.288351 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.289322 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.291305 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.292348 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.292552 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.292679 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.292926 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.293040 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:10:20.293347 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.285875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37747 (local address 127.8.215.2:56683)
0925 05:10:20.287142 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.287158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:20.287332 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:20.287733 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:20.287739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:20.287765 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:20.288167 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:20.288178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.289487 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.289496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.291103 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:20.291124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.292576 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.292581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:20.292699 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.292776 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 05:10:20.292888 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:10:20.293027 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":471,"threads_started":1}
I20260925 05:10:20.293416 23616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c979bc982d45188024e512266e28f8" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e8ef0662d5451e8148c98612a3cdbb" is_pre_election: true
I20260925 05:10:20.294039 23616 raft_consensus.cc:2454] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 0.
I20260925 05:10:20.294500 23879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c979bc982d45188024e512266e28f8" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4f8a2ca50349d488717a917afa86ba" is_pre_election: true
I20260925 05:10:20.294826 23681 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7bde983544438fb63ebba7e7db20c6, 28e8ef0662d5451e8148c98612a3cdbb; no voters:
I20260925 05:10:20.295135 23879 raft_consensus.cc:2454] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 0.
I20260925 05:10:20.295298 23947 raft_consensus.cc:2781] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:20.295485 23947 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:20.295598 23947 raft_consensus.cc:3037] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:20.297941 23947 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.298595 23947 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 election: Requested vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583)
I20260925 05:10:20.299027 23879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c979bc982d45188024e512266e28f8" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4f8a2ca50349d488717a917afa86ba"
I20260925 05:10:20.299239 23879 raft_consensus.cc:3037] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:20.299254 23616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c979bc982d45188024e512266e28f8" candidate_uuid: "1c7bde983544438fb63ebba7e7db20c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e8ef0662d5451e8148c98612a3cdbb"
I20260925 05:10:20.299538 23616 raft_consensus.cc:3037] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:20.301575 23879 raft_consensus.cc:2454] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 1.
I20260925 05:10:20.302026 23683 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba; no voters:
I20260925 05:10:20.302280 23616 raft_consensus.cc:2454] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7bde983544438fb63ebba7e7db20c6 in term 1.
I20260925 05:10:20.302330 23947 raft_consensus.cc:2781] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:20.302850 23947 raft_consensus.cc:686] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7bde983544438fb63ebba7e7db20c6, State: Running, Role: LEADER
I20260925 05:10:20.303411 23947 consensus_queue.cc:237] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:20.308126 23474 catalog_manager.cc:5495] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "1c7bde983544438fb63ebba7e7db20c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:20.368379 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.362958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.8.215.2:36785)
0925 05:10:20.363208 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:20.363215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:20.363233 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:10:20.363459 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:20.363463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:20.363557 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:20.363721 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:20.364102 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.364842 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.366222 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.366633 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.366770 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.367279 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:20.367800 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:20.367959 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.368016 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.368105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 05:10:20.376952 23616 raft_consensus.cc:1261] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Refusing update from remote peer 1c7bde983544438fb63ebba7e7db20c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:20.377045 23879 raft_consensus.cc:1261] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Refusing update from remote peer 1c7bde983544438fb63ebba7e7db20c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:20.377950 23951 consensus_queue.cc:1035] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:20.378304 23947 consensus_queue.cc:1035] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:20.389736 23953 mvcc.cc:205] Tried to move back new op lower bound from 7333122131454267392 to 7333122131167064064. Current Snapshot: MvccSnapshot[applied={T|T < 7333122131454267392}]
I20260925 05:10:20.405432 23474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:20.405764 23474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:20.409824 23474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Sending DeleteTablet for 3 replicas of tablet 16c979bc982d45188024e512266e28f8
I20260925 05:10:20.411051 23859 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c979bc982d45188024e512266e28f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:20 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:54302
I20260925 05:10:20.411229 23728 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c979bc982d45188024e512266e28f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:20 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:50986
I20260925 05:10:20.411273 23596 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c979bc982d45188024e512266e28f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:20 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:45676
I20260925 05:10:20.412307 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 23513
I20260925 05:10:20.429342 23963 tablet_replica.cc:330] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: stopping tablet replica
I20260925 05:10:20.429841 23963 raft_consensus.cc:2229] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:20.430457 23963 raft_consensus.cc:2258] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:20.432947 23963 ts_tablet_manager.cc:1905] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:20.434804 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.413760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.8.215.1:46583)
0925 05:10:20.413976 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:20.413985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:20.414008 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:10:20.414083 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:20.414091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:20.414202 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:20.414349 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:20.416119 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.417675 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.419683 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.420578 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.420744 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.424412 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:20.424436 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:20.424460 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:20.424502 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:20.429159 (+ 4657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:20.430361 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:20.430371 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:20.430374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:20.430571 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:20.431124 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:20.431129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:20.431133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:20.431631 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:20.431942 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.434119 (+ 2177us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.434379 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:10:20.435544 23656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:20.441787 23963 ts_tablet_manager.cc:1918] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:20.441987 23963 log.cc:1196] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/16c979bc982d45188024e512266e28f8
I20260925 05:10:20.442615 23963 ts_tablet_manager.cc:1939] T 16c979bc982d45188024e512266e28f8 P ce4f8a2ca50349d488717a917afa86ba: Deleting consensus metadata
I20260925 05:10:20.444923 23462 catalog_manager.cc:4841] TS ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): tablet 16c979bc982d45188024e512266e28f8 (table test-table [id=9a63169fbba14943a9faccaeb3aedd9a]) successfully deleted
I20260925 05:10:20.447165 23964 tablet_replica.cc:330] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: stopping tablet replica
I20260925 05:10:20.447990 23964 raft_consensus.cc:2229] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:20.449383 23964 raft_consensus.cc:2258] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:20.451776 23964 ts_tablet_manager.cc:1905] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:20.456856 23964 ts_tablet_manager.cc:1918] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:20.457048 23964 log.cc:1196] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/16c979bc982d45188024e512266e28f8
I20260925 05:10:20.457540 23964 ts_tablet_manager.cc:1939] T 16c979bc982d45188024e512266e28f8 P 1c7bde983544438fb63ebba7e7db20c6: Deleting consensus metadata
I20260925 05:10:20.459549 23460 catalog_manager.cc:4841] TS 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): tablet 16c979bc982d45188024e512266e28f8 (table test-table [id=9a63169fbba14943a9faccaeb3aedd9a]) successfully deleted
I20260925 05:10:20.583642 23965 tablet_replica.cc:330] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: stopping tablet replica
I20260925 05:10:20.583647 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 23513
W20260925 05:10:20.602180 23460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.215.1:46583 (error 108)
I20260925 05:10:20.602967 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 23665
I20260925 05:10:20.604667 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.603577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45684)
0925 05:10:20.604325 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.604441 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":529,"threads_started":1}
W20260925 05:10:20.604974 23460 catalog_manager.cc:4584] TS 28e8ef0662d5451e8148c98612a3cdbb (127.8.215.1:46583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16c979bc982d45188024e512266e28f8: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
I20260925 05:10:20.619314 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.603709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.8.215.2:36785)
0925 05:10:20.603954 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:20.603963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:20.604224 (+ 261us) server_negotiation.cc:407] Connection header received
0925 05:10:20.604303 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:20.604308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:20.604399 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:20.604509 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:20.606143 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.607359 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.608630 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.609342 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.609459 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.612066 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:20.612093 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:20.612115 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:20.612165 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:20.616191 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:20.617087 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:20.617098 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:20.617101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:20.617280 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:20.617674 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:20.617680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:20.617683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:20.618161 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:20.618489 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.618657 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.618893 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:10:20.620146 23788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:20.658773 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.658325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45686)
0925 05:10:20.658528 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.658628 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:10:20.659854 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.659462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45690)
0925 05:10:20.659623 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.659714 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 05:10:20.720664 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.720070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45704)
0925 05:10:20.720366 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.720458 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260925 05:10:20.721735 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.721378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45718)
0925 05:10:20.721524 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.721605 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:10:20.735778 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 23665
I20260925 05:10:20.755918 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 23796
I20260925 05:10:20.769675 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.756871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.8.215.3:37747)
0925 05:10:20.757045 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:20.757050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:20.757069 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:10:20.757240 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:20.757244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:20.757308 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:20.757391 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:20.758927 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.759776 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.761174 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:20.761558 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:20.761653 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:20.764254 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:20.764268 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:20.764283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:20.764315 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:20.767169 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:20.767815 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:20.767822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:20.767825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:20.767951 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:20.768277 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:20.768280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:20.768282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:20.768642 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:20.768928 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:20.769137 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 05:10:20.769301 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:10:20.770385 23919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:20.803045 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.802381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45722)
0925 05:10:20.802695 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.802814 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 05:10:20.804211 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.803753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:46583 (local address 127.0.0.1:45734)
0925 05:10:20.803956 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.804049 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 05:10:20.861866 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 23796
I20260925 05:10:20.879134 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 23443
I20260925 05:10:20.880327 23506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:21.001670 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 23443
I20260925 05:10:21.024796 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43921
--webserver_interface=127.8.215.62
--webserver_port=37193
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:21.358057 23972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:21.358337 23972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:21.358505 23972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:21.365696 23972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:21.365813 23972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:21.365914 23972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:21.366019 23972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:21.366101 23972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:21.376231 23972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=37193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:21.377614 23972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:21.378995 23972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:21.390977 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.391014 23978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.391052 23980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:21.391836 23972 server_base.cc:1034] running on GCE node
I20260925 05:10:21.392608 23972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:21.394215 23972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:21.395452 23972 hybrid_clock.cc:648] HybridClock initialized: now 1790313021395384 us; error 29 us; skew 500 ppm
I20260925 05:10:21.395825 23972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:21 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:21.398900 23972 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:21.400447 23972 webserver.cc:460] Webserver started at http://127.8.215.62:37193/ using document root <none> and password file <none>
I20260925 05:10:21.401211 23972 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:21.401360 23972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:21.406682 23972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 05:10:21.409443 23987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:21.410688 23972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:21.410871 23972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "f060c1b53a6741edbe2ab7ae1257e60d"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "3a3765d1e61f940dec2217bf2c83e46d"
server_key_iv: "f046a4ab6943677e8c6e801044fe8e24"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.411824 23972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:21.431666 23972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:21.435973 23972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:21.436257 23972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:21.455328 23972 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43921
I20260925 05:10:21.455363 24038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43921 every 8 connection(s)
I20260925 05:10:21.456938 23972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:21.458585 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 23972
I20260925 05:10:21.459374 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:46583
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:43921
--webserver_port=40739
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:21.463814 24039 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:10:21.466980 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap starting.
I20260925 05:10:21.487084 24039 log.cc:824] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:21.502871 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:21.503429 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d: Bootstrap complete.
I20260925 05:10:21.512498 24039 raft_consensus.cc:348] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:21.513507 24039 raft_consensus.cc:729] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f060c1b53a6741edbe2ab7ae1257e60d, State: Initialized, Role: FOLLOWER
I20260925 05:10:21.514133 24039 consensus_queue.cc:260] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:21.514384 24039 raft_consensus.cc:388] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:21.514525 24039 raft_consensus.cc:482] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:21.514717 24039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:10:21.518911 24039 raft_consensus.cc:504] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:21.519412 24039 leader_election.cc:302] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f060c1b53a6741edbe2ab7ae1257e60d; no voters:
I20260925 05:10:21.520144 24039 leader_election.cc:288] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:10:21.520292 24043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:10:21.521522 24043 raft_consensus.cc:686] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [term 3 LEADER]: Becoming Leader. State: Replica: f060c1b53a6741edbe2ab7ae1257e60d, State: Running, Role: LEADER
I20260925 05:10:21.522154 24043 consensus_queue.cc:237] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } }
I20260925 05:10:21.522430 24039 sys_catalog.cc:564] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:21.524902 24045 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f060c1b53a6741edbe2ab7ae1257e60d. Latest consensus state: current_term: 3 leader_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } } }
I20260925 05:10:21.525236 24045 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:21.525041 24044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060c1b53a6741edbe2ab7ae1257e60d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43921 } } }
I20260925 05:10:21.525345 24044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060c1b53a6741edbe2ab7ae1257e60d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:21.535499 24050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:21.545454 24050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:21.547134 24050 catalog_manager.cc:1259] Loaded cluster ID: 382241bf3e114f0685b6f76ad7060645
I20260925 05:10:21.547336 24050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 05:10:21.549932 24050 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f19bcd079ad google::LogMessage::Fail() at ??:0
@ 0x7f19bcd0a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f19bcd074de google::LogMessage::Flush() at ??:0
I20260925 05:10:21.557379 24060 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f19bcd08a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f19d273a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f19d2737976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f19d27bf610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f19d27bf439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f19bf953345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f19bf95691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f19bf956799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f19bf931eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f19c2c0b6db start_thread at ??:0
@ 0x7f19bbba161f clone at ??:0
W20260925 05:10:21.897742 24041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:21.898010 24041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:21.898108 24041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:21.905764 24041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:21.905872 24041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:21.905988 24041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:10:21.915957 24041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:46583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=40739
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:21.917285 24041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:21.918519 24041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:21.931777 24081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.931862 24080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.931840 24083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:21.934207 24041 server_base.cc:1034] running on GCE node
I20260925 05:10:21.934911 24041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:21.935910 24041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:21.937124 24041 hybrid_clock.cc:648] HybridClock initialized: now 1790313021937063 us; error 39 us; skew 500 ppm
I20260925 05:10:21.937455 24041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:21 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:21.940650 24041 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:10:21.942313 24041 webserver.cc:460] Webserver started at http://127.8.215.1:40739/ using document root <none> and password file <none>
I20260925 05:10:21.943064 24041 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:21.943208 24041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:21.948940 24041 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:10:21.951823 24090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:21.953262 24041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:21.953438 24041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "28e8ef0662d5451e8148c98612a3cdbb"
format_stamp: "Formatted at 2026-09-25 05:10:15 on dist-test-slave-wpdm"
server_key: "f2b6d269f7994de3648d032cc18bee80"
server_key_iv: "58f313ac594633aca093e52c887cfed8"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.954523 24041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:21.984577 24041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:21.988933 24041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:21.989229 24041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:21.990237 24041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:21.992090 24097 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 05:10:21.997866 24041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:10:21.998013 24041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 05:10:21.998152 24041 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:10:22.001394 24041 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:10:22.001502 24041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:22.001848 24097 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Bootstrap starting.
I20260925 05:10:22.029824 24097 log.cc:824] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:22.043566 24097 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:22.044304 24097 tablet_bootstrap.cc:492] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Bootstrap complete.
I20260925 05:10:22.044492 24041 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:46583
I20260925 05:10:22.044536 24204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:46583 every 8 connection(s)
I20260925 05:10:22.045845 24097 ts_tablet_manager.cc:1397] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Time spent bootstrapping tablet: real 0.044s user 0.031s sys 0.011s
I20260925 05:10:22.045848 24041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:22.051724 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.049170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:36159)
0925 05:10:22.050127 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.050444 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
W20260925 05:10:22.052536 24205 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:43921 (0 consecutive failures): Network error: Failed to ping master at 127.8.215.62:43921: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
I20260925 05:10:22.053478 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.053085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:44367)
0925 05:10:22.053260 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.053350 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:10:22.054467 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.054028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:34589)
0925 05:10:22.054196 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.054304 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:10:22.054724 24205 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:10:22.055769 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24041
I20260925 05:10:22.056411 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:36785
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:43921
--webserver_port=34691
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:22.055766 24097 raft_consensus.cc:348] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:22.056787 24097 raft_consensus.cc:729] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8ef0662d5451e8148c98612a3cdbb, State: Initialized, Role: FOLLOWER
I20260925 05:10:22.057456 24097 consensus_queue.cc:260] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:22.059185 24097 ts_tablet_manager.cc:1428] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 05:10:22.059207 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.058757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:44833)
0925 05:10:22.058951 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.059047 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:10:22.060161 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.059832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:40871)
0925 05:10:22.059971 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.060044 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:10:22.389201 24209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:22.389485 24209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:22.389578 24209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:22.396709 24209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:22.396847 24209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:22.396968 24209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:10:22.406620 24209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:36785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=34691
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:22.407934 24209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:22.409225 24209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:22.422428 24214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.422477 24215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.422541 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:22.423759 24209 server_base.cc:1034] running on GCE node
I20260925 05:10:22.424430 24209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:22.425544 24209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:22.426790 24209 hybrid_clock.cc:648] HybridClock initialized: now 1790313022426738 us; error 45 us; skew 500 ppm
I20260925 05:10:22.427135 24209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:22 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313022, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:22.430469 24209 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:10:22.431973 24209 webserver.cc:460] Webserver started at http://127.8.215.2:34691/ using document root <none> and password file <none>
I20260925 05:10:22.432699 24209 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:22.432893 24209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:22.438789 24209 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:10:22.441723 24224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.443186 24209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:22.443367 24209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "1c7bde983544438fb63ebba7e7db20c6"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "20134d3bfcc48253c04d589c5ca4919f"
server_key_iv: "1c1cbdc3772a8960a155b23786ae4bd9"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.444393 24209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:22.486186 24209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:22.490610 24209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:22.490885 24209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:22.491923 24209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:22.493330 24209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:22.493456 24209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.493583 24209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:22.493670 24209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.527594 24209 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:36785
I20260925 05:10:22.527648 24336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:36785 every 8 connection(s)
I20260925 05:10:22.529114 24209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:22.531380 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24209
I20260925 05:10:22.532083 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:37747
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:43921
--webserver_port=45983
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:22.534538 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.532080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:38567)
0925 05:10:22.533043 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.533331 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":319,"threads_started":1}
W20260925 05:10:22.535306 24337 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:43921 (0 consecutive failures): Network error: Failed to ping master at 127.8.215.62:43921: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
I20260925 05:10:22.536269 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.535829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:46357)
0925 05:10:22.536013 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.536111 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:10:22.537356 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.536906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:52579)
0925 05:10:22.537074 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.537184 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:10:22.537649 24337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:10:22.857394 24340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:22.857677 24340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:22.857890 24340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:22.864867 24340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:22.864964 24340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:22.865070 24340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:10:22.874203 24340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:37747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=45983
--tserver_master_addrs=127.8.215.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:22.875423 24340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:22.876641 24340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:22.889410 24345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.889326 24346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.889381 24348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:22.890357 24340 server_base.cc:1034] running on GCE node
I20260925 05:10:22.891007 24340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:22.891965 24340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:22.893184 24340 hybrid_clock.cc:648] HybridClock initialized: now 1790313022893139 us; error 56 us; skew 500 ppm
I20260925 05:10:22.893498 24340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:22 dist-test-slave-wpdm krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313022, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:22.896395 24340 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:10:22.897908 24340 webserver.cc:460] Webserver started at http://127.8.215.3:45983/ using document root <none> and password file <none>
I20260925 05:10:22.898581 24340 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:22.898722 24340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:22.904034 24340 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 05:10:22.906719 24355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.908031 24340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:22.908211 24340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "ce4f8a2ca50349d488717a917afa86ba"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-wpdm"
server_key: "37e0772b14b355d332dddd814a11a0aa"
server_key_iv: "bb9459727f4828b424ac200c8c2c6f4d"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.909276 24340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:22.941875 24340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:22.946125 24340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:22.946409 24340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:22.947399 24340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:22.948947 24340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:22.949061 24340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.949186 24340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:22.949286 24340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.983057 24340 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:37747
I20260925 05:10:22.983161 24467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:37747 every 8 connection(s)
I20260925 05:10:22.984562 24340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:22.988940 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24340
I20260925 05:10:22.989764 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.987591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:40673)
0925 05:10:22.988386 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.988632 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":276,"threads_started":1}
W20260925 05:10:22.990590 24468 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:43921 (0 consecutive failures): Network error: Failed to ping master at 127.8.215.62:43921: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
I20260925 05:10:22.991398 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.991088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:38941)
0925 05:10:22.991234 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.991297 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 05:10:22.992237 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.991889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:42977)
0925 05:10:22.992037 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.992130 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 05:10:22.992502 24468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:10:23.061779 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.061024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:55231)
0925 05:10:23.061490 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:23.061576 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260925 05:10:23.246387 24473 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:23.246696 24473 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:23.248498 24473 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
Sep 25 05:10:23 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
Sep 25 05:10:23 dist-test-slave-wpdm krb5kdc[22890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:10:23.270272 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.249843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:53055 (local address 127.8.215.3:37747)
0925 05:10:23.251387 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:23.251396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:23.251439 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:10:23.252877 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:23.252885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:23.253193 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:23.253505 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:23.255073 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.256223 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.258044 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.258737 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.258999 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:23.263507 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:23.263515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:23.263528 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:23.263605 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:23.266464 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:23.267329 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:23.267336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:23.267342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:23.267496 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:23.267937 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:23.267941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:23.267946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:23.268465 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:23.268670 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:23.269488 (+ 818us) server_negotiation.cc:299] Negotiation successful
0925 05:10:23.269863 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":1194,"threads_started":1}
I20260925 05:10:23.270278 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.249402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:32867 (local address 127.8.215.2:36785)
0925 05:10:23.250713 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:23.250732 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:23.250783 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:10:23.252393 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:23.252402 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:23.252840 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:23.253165 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:23.255062 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.256221 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.257984 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.258690 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.258892 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:23.263189 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:23.263197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:23.263221 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:23.263258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:23.266444 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:23.267320 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:23.267328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:23.267333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:23.267487 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:23.267938 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:23.267942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:23.267946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:23.268465 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:23.268668 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:23.269487 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 05:10:23.269836 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":922,"threads_started":1}
I20260925 05:10:23.270295 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.248905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:37747 (local address 127.8.215.1:53055)
0925 05:10:23.251001 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:23.251033 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:23.251158 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:23.253438 (+ 2280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:23.253444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:23.254179 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:23.254867 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:23.254886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.256408 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.256412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:23.257859 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:23.257870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.258945 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.258951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:23.259191 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:23.260070 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:23.260087 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:23.263227 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:23.266619 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:23.266625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:23.266629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:23.267176 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:23.267623 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:23.267630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:23.267634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:23.267820 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:23.268611 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:23.268619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:23.269007 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 05:10:23.269251 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 05:10:23.269811 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"mutex_wait_us":149,"spinlock_wait_cycles":17792,"thread_start_us":1803,"threads_started":1}
I20260925 05:10:23.270351 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.248761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36785 (local address 127.8.215.1:32867)
0925 05:10:23.249129 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:23.249166 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:23.249735 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:23.253299 (+ 3564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:23.253312 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:23.254145 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:23.254868 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:23.254887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.256416 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.256420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:23.257829 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:23.257841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.258889 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.258896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:23.259188 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:23.260001 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:23.260027 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:23.262864 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:23.266597 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:23.266607 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:23.266622 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:23.267169 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:23.267602 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:23.267607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:23.267610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:23.267817 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:23.268610 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:23.268618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:23.268987 (+ 369us) client_negotiation.cc:769] Sending connection context
0925 05:10:23.269252 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 05:10:23.269948 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":470}
W20260925 05:10:23.274132 24093 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
W20260925 05:10:23.274451 24091 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
I20260925 05:10:23.274588 24091 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28e8ef0662d5451e8148c98612a3cdbb; no voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba
I20260925 05:10:23.274947 24473 raft_consensus.cc:2726] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:10:23.539119 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.538192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:40813)
0925 05:10:23.538702 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:23.538842 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":252,"threads_started":1}
I20260925 05:10:23.993760 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.993038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:51011)
0925 05:10:23.993455 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:23.993559 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":205,"threads_started":1}
I20260925 05:10:24.063548 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:24.062707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:39677)
0925 05:10:24.063254 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:24.063357 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":279,"threads_started":1}
I20260925 05:10:24.540978 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:24.540091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:50569)
0925 05:10:24.540644 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:24.540761 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":283,"threads_started":1}
I20260925 05:10:24.995276 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:24.994616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:41835)
0925 05:10:24.995022 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:24.995100 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260925 05:10:25.065371 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.064496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:41877)
0925 05:10:25.065087 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:25.065173 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":303,"threads_started":1}
I20260925 05:10:25.235925 24483 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:25.236169 24483 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:25.236981 24483 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
W20260925 05:10:25.238049 24091 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
W20260925 05:10:25.238236 24093 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
I20260925 05:10:25.238348 24093 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28e8ef0662d5451e8148c98612a3cdbb; no voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba
I20260925 05:10:25.238588 24483 raft_consensus.cc:2726] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:10:25.542655 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.541865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:56091)
0925 05:10:25.542410 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:25.542491 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":308,"threads_started":1}
I20260925 05:10:25.996857 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.996137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:35907)
0925 05:10:25.996547 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:25.996634 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260925 05:10:26.067107 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:26.066290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:39499)
0925 05:10:26.066873 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:26.066951 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":371,"threads_started":1}
I20260925 05:10:26.544294 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:26.543585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:46321)
0925 05:10:26.544034 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:26.544117 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":221,"threads_started":1}
I20260925 05:10:26.998533 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:26.997838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:54629)
0925 05:10:26.998277 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:26.998364 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260925 05:10:27.069013 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.068286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:41359)
0925 05:10:27.068688 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:27.068837 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
I20260925 05:10:27.546000 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.545309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:53715)
0925 05:10:27.545771 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:27.545847 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260925 05:10:28.000180 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.999452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:32777)
0925 05:10:27.999924 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:28.000005 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
I20260925 05:10:28.070549 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:28.069861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:42497)
0925 05:10:28.070301 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:28.070382 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260925 05:10:28.468765 24493 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:28.469097 24493 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:28.469873 24493 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
W20260925 05:10:28.470847 24091 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
W20260925 05:10:28.470980 24093 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
I20260925 05:10:28.471161 24093 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28e8ef0662d5451e8148c98612a3cdbb; no voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba
I20260925 05:10:28.471431 24493 raft_consensus.cc:2726] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:10:28.547885 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:28.546930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:42105)
0925 05:10:28.547424 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:28.547507 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260925 05:10:29.001904 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:29.001175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:55191)
0925 05:10:29.001655 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:29.001734 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260925 05:10:29.072342 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:29.071609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:50611)
0925 05:10:29.072045 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:29.072130 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":237,"threads_started":1}
I20260925 05:10:29.549868 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:29.548911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:58811)
0925 05:10:29.549559 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:29.549639 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260925 05:10:30.003789 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:30.002890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:35197)
0925 05:10:30.003520 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:30.003594 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":325,"threads_started":1}
I20260925 05:10:30.073962 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:30.073309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:42859)
0925 05:10:30.073725 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:30.073808 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260925 05:10:30.551618 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:30.550863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:36329)
0925 05:10:30.551292 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:30.551380 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260925 05:10:31.005558 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:31.004813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:44449)
0925 05:10:31.005298 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:31.005385 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260925 05:10:31.075584 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:31.074870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:51537)
0925 05:10:31.075307 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:31.075395 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260925 05:10:31.553375 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:31.552570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:45937)
0925 05:10:31.553112 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:31.553216 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260925 05:10:31.621191 24504 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:31.621451 24504 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:31.622129 24504 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
W20260925 05:10:31.623133 24091 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
W20260925 05:10:31.623304 24093 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
I20260925 05:10:31.623467 24093 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28e8ef0662d5451e8148c98612a3cdbb; no voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba
I20260925 05:10:31.623720 24504 raft_consensus.cc:2726] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:10:32.007372 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.006589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:53453)
0925 05:10:32.007113 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.007208 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":230,"threads_started":1}
I20260925 05:10:32.077771 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.076781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:57081)
0925 05:10:32.077374 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.077530 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":305,"threads_started":1}
I20260925 05:10:32.555160 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.554396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:34785)
0925 05:10:32.554850 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.554937 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260925 05:10:33.009177 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.008387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:58577)
0925 05:10:33.008921 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.009000 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":281,"threads_started":1}
I20260925 05:10:33.079419 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.078671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:50057)
0925 05:10:33.079159 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.079243 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I20260925 05:10:33.556984 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.556161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:51221)
0925 05:10:33.556636 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.556821 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260925 05:10:34.010986 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.010195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:36903)
0925 05:10:34.010734 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:34.010827 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
I20260925 05:10:34.081207 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.080381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:55635)
0925 05:10:34.080919 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:34.081008 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":261,"threads_started":1}
I20260925 05:10:34.558640 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.557901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:32783)
0925 05:10:34.558347 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:34.558424 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260925 05:10:35.012904 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:35.011959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:38461)
0925 05:10:35.012552 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:35.012630 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260925 05:10:35.082839 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:35.082111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:58983)
0925 05:10:35.082526 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:35.082662 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":224,"threads_started":1}
I20260925 05:10:35.560359 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:35.559604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:53957)
0925 05:10:35.560060 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:35.560160 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260925 05:10:36.014575 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.013748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:57179)
0925 05:10:36.014276 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:36.014376 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
I20260925 05:10:36.084450 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.083765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:57045)
0925 05:10:36.084191 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:36.084271 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260925 05:10:36.562033 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.561312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:48981)
0925 05:10:36.561756 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:36.561839 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260925 05:10:36.569152 24520 raft_consensus.cc:482] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:36.569392 24520 raft_consensus.cc:504] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4f8a2ca50349d488717a917afa86ba" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 37747 } } peers { permanent_uuid: "1c7bde983544438fb63ebba7e7db20c6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36785 } } peers { permanent_uuid: "28e8ef0662d5451e8148c98612a3cdbb" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 46583 } }
I20260925 05:10:36.570097 24520 leader_election.cc:288] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747), 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785)
W20260925 05:10:36.571069 24091 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c7bde983544438fb63ebba7e7db20c6 (127.8.215.2:36785): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
W20260925 05:10:36.571350 24093 leader_election.cc:341] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce4f8a2ca50349d488717a917afa86ba (127.8.215.3:37747): Not found: Tablet not found: 16c979bc982d45188024e512266e28f8
I20260925 05:10:36.571478 24093 leader_election.cc:302] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28e8ef0662d5451e8148c98612a3cdbb; no voters: 1c7bde983544438fb63ebba7e7db20c6, ce4f8a2ca50349d488717a917afa86ba
I20260925 05:10:36.571693 24520 raft_consensus.cc:2726] T 16c979bc982d45188024e512266e28f8 P 28e8ef0662d5451e8148c98612a3cdbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:10:37.016399 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:37.015462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:59391)
0925 05:10:37.016079 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:37.016198 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260925 05:10:37.086323 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:37.085668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:55715)
0925 05:10:37.086055 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:37.086139 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":195,"threads_started":1}
I20260925 05:10:37.564078 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:37.563351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.2:41155)
0925 05:10:37.563796 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:37.563911 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260925 05:10:38.001510 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24041
I20260925 05:10:38.018268 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.017501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.3:34509)
0925 05:10:38.017975 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.018074 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260925 05:10:38.088294 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.087376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43921 (local address 127.8.215.1:39401)
0925 05:10:38.087982 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.088101 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":291,"threads_started":1}
I20260925 05:10:38.416344 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.002752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.8.215.1:46583)
0925 05:10:38.003369 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.003398 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.401610 (+398212us) server_negotiation.cc:407] Connection header received
0925 05:10:38.401831 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.401840 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.402092 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.402275 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.403768 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.404880 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.406155 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.406757 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.406903 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.409683 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.409695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.409711 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.409749 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.413051 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.413892 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.413909 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.413912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.414075 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.414524 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.414531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.414535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.414960 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.415336 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.415676 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.415979 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
I20260925 05:10:38.417757 24200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:38.509337 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24041
I20260925 05:10:38.529431 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24209
I20260925 05:10:38.545675 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.530275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.8.215.2:36785)
0925 05:10:38.533117 (+ 2842us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.533122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.533138 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:10:38.533201 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.533205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.533262 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.533407 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.534941 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.535741 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.537092 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.537546 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.537687 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.540366 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.540371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.540375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.540409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.542945 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.543623 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.543633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.543635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.543770 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.544152 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.544163 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.544165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.544444 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.544731 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.545054 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.545301 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2738,"thread_start_us":2599,"threads_started":1}
I20260925 05:10:38.546404 24332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:38.634925 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24209
I20260925 05:10:38.654060 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24340
I20260925 05:10:38.667826 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.654842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.8.215.3:37747)
0925 05:10:38.655370 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.655375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.655393 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:10:38.655448 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.655452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.655508 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.655623 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.657052 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.657886 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.659091 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.659507 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.659602 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.662293 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.662297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.662301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.662335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.665338 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.666039 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.666043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.666046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.666185 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.666597 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.666601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.666603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.666886 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.667135 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.667351 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.667527 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260925 05:10:38.668599 24463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:38.754218 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24340
2026-09-25T05:10:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25305 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:38 dist-test-slave-wpdm krb5kdc[24535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:38 dist-test-slave-wpdm krb5kdc[24535](info): set up 2 sockets
Sep 25 05:10:38 dist-test-slave-wpdm krb5kdc[24535](info): commencing operation
krb5kdc: starting...
W20260925 05:10:39.849763 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:10:40 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:10:40.104565 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44205
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:40.441721 24548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:40.442013 24548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:40.442188 24548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:40.449558 24548 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:10:40.449657 24548 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:10:40.449743 24548 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:10:40.449848 24548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:40.449944 24548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:40.450062 24548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:40.450140 24548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:40.450220 24548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:40.460335 24548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:40.461678 24548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:40.463042 24548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:40.475497 24556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:40.475553 24554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:40.475452 24553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:40.482455 24548 server_base.cc:1034] running on GCE node
I20260925 05:10:40.483333 24548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:40.484886 24548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:40.486136 24548 hybrid_clock.cc:648] HybridClock initialized: now 1790313040486076 us; error 45 us; skew 500 ppm
I20260925 05:10:40.486480 24548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:40 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:40.489787 24548 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:10:40.491310 24548 webserver.cc:460] Webserver started at http://127.8.215.62:40091/ using document root <none> and password file <none>
I20260925 05:10:40.492054 24548 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:40.492187 24548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:40.492504 24548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:40.495589 24548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "2c5f9884285b41b3ae5f1ab53572566d"
format_stamp: "Formatted at 2026-09-25 05:10:40 on dist-test-slave-wpdm"
server_key: "3aca3280dcb72425c1954daf8cc6c3ea"
server_key_iv: "a34229233a0803b5d5bbf23427b571c3"
server_key_version: "encryptionkey@0"
I20260925 05:10:40.496374 24548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "2c5f9884285b41b3ae5f1ab53572566d"
format_stamp: "Formatted at 2026-09-25 05:10:40 on dist-test-slave-wpdm"
server_key: "3aca3280dcb72425c1954daf8cc6c3ea"
server_key_iv: "a34229233a0803b5d5bbf23427b571c3"
server_key_version: "encryptionkey@0"
I20260925 05:10:40.501672 24548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:10:40.505215 24563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:40.506559 24548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 05:10:40.506747 24548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "2c5f9884285b41b3ae5f1ab53572566d"
format_stamp: "Formatted at 2026-09-25 05:10:40 on dist-test-slave-wpdm"
server_key: "3aca3280dcb72425c1954daf8cc6c3ea"
server_key_iv: "a34229233a0803b5d5bbf23427b571c3"
server_key_version: "encryptionkey@0"
I20260925 05:10:40.506974 24548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:40.525549 24548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:40.530141 24548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:40.530427 24548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:40.549448 24548 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:44205
I20260925 05:10:40.549546 24614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:44205 every 8 connection(s)
I20260925 05:10:40.550963 24548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:40.555009 24615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:40.560122 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24548
I20260925 05:10:40.560386 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:10:40.561226 9052 external_mini_cluster.cc:1428] Setting key 10e018aaf69d0e0febbf6785a6ece9c0
I20260925 05:10:40.567400 24615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap starting.
I20260925 05:10:40.571381 24615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:40.572631 24615 log.cc:824] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:40.576304 24615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: No bootstrap required, opened a new log
Sep 25 05:10:40 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:40.587345 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:40.563381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.8.215.62:44205)
0925 05:10:40.564331 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:40.564346 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:40.564390 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:10:40.566346 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:40.566373 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:40.566810 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:40.567166 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:40.569082 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:40.570432 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:40.572321 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:40.573156 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:40.573391 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:40.578258 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:40.578270 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:40.578292 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:40.578348 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:40.582524 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:40.583245 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:40.583255 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:40.583267 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:40.583487 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:40.583866 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:40.583871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:40.583875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:40.584645 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:40.584973 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:40.585422 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 05:10:40.585918 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":493,"threads_started":1}
I20260925 05:10:40.587283 24615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:10:40.587783 24615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:40.587920 24615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5f9884285b41b3ae5f1ab53572566d, State: Initialized, Role: FOLLOWER
I20260925 05:10:40.588490 24615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:10:40.588766 24615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:40.588943 24615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:40.589135 24615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:40.591485 24615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:10:40.591982 24615 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5f9884285b41b3ae5f1ab53572566d; no voters:
I20260925 05:10:40.592535 24615 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:40.592687 24620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:40.593469 24620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 LEADER]: Becoming Leader. State: Replica: 2c5f9884285b41b3ae5f1ab53572566d, State: Running, Role: LEADER
I20260925 05:10:40.594053 24620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:10:40.594686 24615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:40.596688 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c5f9884285b41b3ae5f1ab53572566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } } }
I20260925 05:10:40.597093 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:40.597604 24622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5f9884285b41b3ae5f1ab53572566d. Latest consensus state: current_term: 1 leader_uuid: "2c5f9884285b41b3ae5f1ab53572566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } } }
I20260925 05:10:40.597919 24622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:40.598174 24627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:40.604939 24627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:40.616086 24627 catalog_manager.cc:1347] Generated new cluster ID: 6cb57907b32f48c39769df240b3315e6
I20260925 05:10:40.616212 24627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:40.651407 24627 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:40.652971 24627 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:40.659390 24627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Generated new TSK 0
I20260925 05:10:40.660238 24627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:10:40.722885 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44205
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:41.081271 24641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:41.081615 24641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:41.081791 24641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:41.089413 24641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:41.089570 24641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:41.089740 24641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:10:41.100306 24641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:41.101686 24641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:41.103004 24641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:41.116967 24647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:41.116917 24646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:41.117002 24649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:41.117825 24641 server_base.cc:1034] running on GCE node
I20260925 05:10:41.118445 24641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:41.119424 24641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:41.120664 24641 hybrid_clock.cc:648] HybridClock initialized: now 1790313041120616 us; error 57 us; skew 500 ppm
I20260925 05:10:41.121017 24641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:41 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:41.127929 24641 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:10:41.129541 24641 webserver.cc:460] Webserver started at http://127.8.215.1:33867/ using document root <none> and password file <none>
I20260925 05:10:41.130257 24641 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:41.130411 24641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:41.130729 24641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:41.133755 24641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "60bf15a821154e87b3509ae851ae77a7"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "9600098b6d49da5c1f6eb1275309dc23"
server_key_iv: "5d4f6a25c2c19280e9cef23d83d7fee5"
server_key_version: "encryptionkey@0"
I20260925 05:10:41.134516 24641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "60bf15a821154e87b3509ae851ae77a7"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "9600098b6d49da5c1f6eb1275309dc23"
server_key_iv: "5d4f6a25c2c19280e9cef23d83d7fee5"
server_key_version: "encryptionkey@0"
I20260925 05:10:41.139570 24641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:10:41.143007 24656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:41.144268 24641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:10:41.144445 24641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "60bf15a821154e87b3509ae851ae77a7"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "9600098b6d49da5c1f6eb1275309dc23"
server_key_iv: "5d4f6a25c2c19280e9cef23d83d7fee5"
server_key_version: "encryptionkey@0"
I20260925 05:10:41.144649 24641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:41.179641 24641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:41.184352 24641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:41.184664 24641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:41.185688 24641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:41.187034 24641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:41.187145 24641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:41.187311 24641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:41.187388 24641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:41.221194 24641 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:37367
I20260925 05:10:41.221280 24768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:37367 every 8 connection(s)
I20260925 05:10:41.222506 24641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:41.231938 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24641
I20260925 05:10:41.232178 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:41.232872 9052 external_mini_cluster.cc:1428] Setting key bc2a23a14763f07635449b0d7923f609
Sep 25 05:10:41 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:41.242686 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.225177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:48315 (local address 127.8.215.62:44205)
0925 05:10:41.225647 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:41.225654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:41.226487 (+ 833us) server_negotiation.cc:407] Connection header received
0925 05:10:41.228314 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:41.228320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:41.228412 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:41.228535 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:41.230100 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.230844 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.232522 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.233036 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.233179 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.237011 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:41.237016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:41.237019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:41.237049 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:41.239591 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:41.240303 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:41.240311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:41.240317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:41.240440 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:41.240919 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:41.240925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:41.240928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:41.241203 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:41.241359 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:41.242080 (+ 721us) server_negotiation.cc:299] Negotiation successful
0925 05:10:41.242320 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":252,"threads_started":1}
I20260925 05:10:41.243671 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.225637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:48315)
0925 05:10:41.226324 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.226361 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:41.228042 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:41.228843 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:41.228852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:41.229364 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:41.229925 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.229942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.231003 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.231016 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.232383 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.232396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.233281 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.233290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.233598 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.234339 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:41.234373 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:41.236799 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:41.239741 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:41.239750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:41.239768 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:41.240183 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:41.240552 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:41.240557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:41.240561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:41.240802 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:41.241332 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:41.241340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:41.241701 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 05:10:41.241965 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 05:10:41.242498 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":275,"threads_started":1}
I20260925 05:10:41.245334 24769 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44205
I20260925 05:10:41.245637 24769 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:41.246357 24769 heartbeater.cc:511] Master 127.8.215.62:44205 requested a full tablet report, sending...
I20260925 05:10:41.248484 24580 ts_manager.cc:194] Registered new tserver with Master: 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367)
I20260925 05:10:41.249987 24580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:48315
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:10:41.289588 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44205
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:41.622422 24775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:41.622711 24775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:41.622807 24775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:41.630151 24775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:41.630260 24775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:41.630373 24775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:10:41.640115 24775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:41.641348 24775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:41.642709 24775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:41.655933 24783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:41.656064 24780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:41.656018 24781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:41.657363 24775 server_base.cc:1034] running on GCE node
I20260925 05:10:41.658046 24775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:41.658969 24775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:41.660185 24775 hybrid_clock.cc:648] HybridClock initialized: now 1790313041660129 us; error 39 us; skew 500 ppm
I20260925 05:10:41.660552 24775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:41 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:41.663844 24775 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:10:41.665422 24775 webserver.cc:460] Webserver started at http://127.8.215.2:39995/ using document root <none> and password file <none>
I20260925 05:10:41.666160 24775 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:41.666293 24775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:41.666623 24775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:41.669772 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "0e59a88f933536b08a9bb62ea7258632"
server_key_iv: "333ebbb9c3dfe7cbc661f165bad02edf"
server_key_version: "encryptionkey@0"
I20260925 05:10:41.670538 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "0e59a88f933536b08a9bb62ea7258632"
server_key_iv: "333ebbb9c3dfe7cbc661f165bad02edf"
server_key_version: "encryptionkey@0"
I20260925 05:10:41.675601 24775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:10:41.679049 24790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:41.680347 24775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:10:41.680518 24775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "0e59a88f933536b08a9bb62ea7258632"
server_key_iv: "333ebbb9c3dfe7cbc661f165bad02edf"
server_key_version: "encryptionkey@0"
I20260925 05:10:41.680727 24775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:41.698699 24775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:41.703013 24775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:41.703320 24775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:41.704257 24775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:41.705679 24775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:41.705750 24775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:41.705909 24775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:41.705960 24775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:41.740094 24775 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:44793
I20260925 05:10:41.740204 24902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:44793 every 8 connection(s)
I20260925 05:10:41.741453 24775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:41.744360 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24775
I20260925 05:10:41.744576 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:41.745136 9052 external_mini_cluster.cc:1428] Setting key 247382a5b91f1c9aa0b19c048d0fac18
Sep 25 05:10:41 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:41.764202 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.744210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:60945 (local address 127.8.215.62:44205)
0925 05:10:41.744714 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:41.744751 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:41.746171 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 05:10:41.747865 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:41.747873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:41.747962 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:41.748074 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:41.749911 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.751069 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.752605 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.753399 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.753543 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.757092 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:41.757102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:41.757110 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:41.757158 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:41.760786 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:41.761520 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:41.761535 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:41.761541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:41.761706 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:41.762117 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:41.762125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:41.762131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:41.762568 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:41.762719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:41.763405 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 05:10:41.763695 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260925 05:10:41.764830 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.744909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:60945)
0925 05:10:41.745771 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.745824 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:41.747588 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:41.748371 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:41.748380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:41.749024 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:41.749737 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.749757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.751217 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.751221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.752454 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.752465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.753616 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.753621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.753825 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.754560 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:41.754588 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:41.756856 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:41.760922 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:41.760929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:41.760944 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:41.761384 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:41.761813 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:41.761816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:41.761819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:41.761996 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:41.762682 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:41.762688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:41.763001 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 05:10:41.763238 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 05:10:41.763623 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":349,"threads_started":1}
I20260925 05:10:41.766368 24903 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44205
I20260925 05:10:41.766681 24903 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:41.767421 24903 heartbeater.cc:511] Master 127.8.215.62:44205 requested a full tablet report, sending...
I20260925 05:10:41.769081 24580 ts_manager.cc:194] Registered new tserver with Master: ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2:44793)
I20260925 05:10:41.769906 24580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:60945
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:10:41.802301 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44205
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:42.135694 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:42.135980 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:42.136158 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:42.143378 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:42.143532 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:42.143723 24909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:10:42.153872 24909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:10:42.155246 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:42.156539 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:42.169638 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:42.169734 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:42.169694 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:42.171379 24909 server_base.cc:1034] running on GCE node
I20260925 05:10:42.171995 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:42.173023 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:42.174249 24909 hybrid_clock.cc:648] HybridClock initialized: now 1790313042174220 us; error 57 us; skew 500 ppm
I20260925 05:10:42.174579 24909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:10:42 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:42.177718 24909 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:10:42.179172 24909 webserver.cc:460] Webserver started at http://127.8.215.3:37201/ using document root <none> and password file <none>
I20260925 05:10:42.179836 24909 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:42.179929 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:42.180186 24909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:42.183109 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "7571025ed4cd4a13abe5ca860a781e8d"
format_stamp: "Formatted at 2026-09-25 05:10:42 on dist-test-slave-wpdm"
server_key: "368818a203fbd8ef2910007a2ed90aa7"
server_key_iv: "645c4694093dd0c04bbf2cec0b4ff9de"
server_key_version: "encryptionkey@0"
I20260925 05:10:42.183856 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "7571025ed4cd4a13abe5ca860a781e8d"
format_stamp: "Formatted at 2026-09-25 05:10:42 on dist-test-slave-wpdm"
server_key: "368818a203fbd8ef2910007a2ed90aa7"
server_key_iv: "645c4694093dd0c04bbf2cec0b4ff9de"
server_key_version: "encryptionkey@0"
I20260925 05:10:42.189199 24909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:10:42.192579 24924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:42.194020 24909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:42.194214 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "7571025ed4cd4a13abe5ca860a781e8d"
format_stamp: "Formatted at 2026-09-25 05:10:42 on dist-test-slave-wpdm"
server_key: "368818a203fbd8ef2910007a2ed90aa7"
server_key_iv: "645c4694093dd0c04bbf2cec0b4ff9de"
server_key_version: "encryptionkey@0"
I20260925 05:10:42.194422 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:42.210397 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:42.214509 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:42.214807 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:42.215790 24909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:42.217134 24909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:42.217211 24909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:42.217394 24909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:42.217446 24909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:42.251554 24909 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:45163
I20260925 05:10:42.251588 25036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:45163 every 8 connection(s)
I20260925 05:10:42.252933 24769 heartbeater.cc:503] Master 127.8.215.62:44205 was elected leader, sending a full tablet report...
I20260925 05:10:42.253019 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:42.257134 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 24909
I20260925 05:10:42.257339 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:42.257841 9052 external_mini_cluster.cc:1428] Setting key 1ca2328829d1f2c5033a2a5004f3208d
Sep 25 05:10:42 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:10:42.273676 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.255632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:49539 (local address 127.8.215.62:44205)
0925 05:10:42.255859 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.255866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.257272 (+ 1406us) server_negotiation.cc:407] Connection header received
0925 05:10:42.259369 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.259374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.259468 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.259581 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:42.261418 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.262154 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.263701 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.264106 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.264244 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.267681 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:42.267687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:42.267690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:42.267722 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:42.270412 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.271146 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.271151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.271154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.271270 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.271700 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.271706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.271714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.272024 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:42.272184 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.273110 (+ 926us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.273330 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:10:42.274652 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.256231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:49539)
0925 05:10:42.257112 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.257157 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:42.259105 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:42.259814 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:42.259822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:42.260581 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:42.261248 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.261270 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.262308 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.262314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.263558 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.263569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.264332 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.264340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.264573 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.265248 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:42.265270 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:42.267451 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:42.270557 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:42.270569 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:42.270586 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:42.271003 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:42.271380 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:42.271384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:42.271388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:42.271575 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:42.272153 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:42.272161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:42.272598 (+ 437us) client_negotiation.cc:769] Sending connection context
0925 05:10:42.272990 (+ 392us) client_negotiation.cc:240] Negotiation successful
0925 05:10:42.273542 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":370,"threads_started":1}
I20260925 05:10:42.275934 25037 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44205
I20260925 05:10:42.276217 25037 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:42.276978 25037 heartbeater.cc:511] Master 127.8.215.62:44205 requested a full tablet report, sending...
I20260925 05:10:42.278339 24580 ts_manager.cc:194] Registered new tserver with Master: 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3:45163)
I20260925 05:10:42.279136 24580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:49539
I20260925 05:10:42.285885 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:42.302843 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.290203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.8.215.62:44205)
0925 05:10:42.290373 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.290378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.290824 (+ 446us) server_negotiation.cc:407] Connection header received
0925 05:10:42.290960 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.290963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.291037 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.291119 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:42.292556 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.293348 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.294863 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.295306 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.295402 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.297858 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:42.297862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:42.297865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:42.297913 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:42.300520 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.301225 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.301229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.301231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.301351 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.301668 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.301671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.301673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.301970 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:42.302161 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.302427 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.302594 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:10:42.306819 24580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:42.313894 24580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:42.338955 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.326524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:45163 (local address 127.0.0.1:42892)
0925 05:10:42.328643 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.328682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:42.328908 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:42.330652 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:42.330663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:42.330697 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:42.331157 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.331171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.333249 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.333257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.335910 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.335925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.337817 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.337823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.338010 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.338066 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 05:10:42.338210 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 05:10:42.338396 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1922,"mutex_wait_us":62,"thread_start_us":1753,"threads_started":1}
I20260925 05:10:42.339807 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.327755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37367 (local address 127.0.0.1:41094)
0925 05:10:42.331462 (+ 3707us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.331482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:42.331606 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:42.332617 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:42.332622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:42.332646 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:42.333099 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.333109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.335133 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.335139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.337089 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.337104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.338904 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.338913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.339114 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.339190 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 05:10:42.339254 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:10:42.339387 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"mutex_wait_us":34,"thread_start_us":1467,"threads_started":1}
I20260925 05:10:42.339869 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.329692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.8.215.2:44793)
0925 05:10:42.330176 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.330184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.332123 (+ 1939us) server_negotiation.cc:407] Connection header received
0925 05:10:42.332283 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.332288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.332462 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.332654 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:42.333513 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.334734 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.337134 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.338378 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.338493 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.338681 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.339382 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.339561 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":254,"threads_started":1}
I20260925 05:10:42.339936 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.326935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.8.215.3:45163)
0925 05:10:42.327891 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.327904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.328817 (+ 913us) server_negotiation.cc:407] Connection header received
0925 05:10:42.329129 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.329141 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.329432 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.330290 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:42.331401 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.333104 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.336082 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.337613 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.337753 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.337995 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.338270 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.339583 (+ 1313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":590,"threads_started":1}
I20260925 05:10:42.340575 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.327650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.8.215.1:37367)
0925 05:10:42.328318 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.328328 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.332025 (+ 3697us) server_negotiation.cc:407] Connection header received
0925 05:10:42.332133 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.332154 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.332409 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.332602 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:42.333321 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.334963 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.337231 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.338609 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.339724 (+ 1115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.339951 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.340050 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.340220 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":375,"threads_started":1}
I20260925 05:10:42.340621 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.329367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:44793 (local address 127.0.0.1:36584)
0925 05:10:42.331998 (+ 2631us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.332014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:42.332143 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:42.332868 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:42.332873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:42.332897 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:42.333349 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.333359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.334895 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.334902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.336969 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.336983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.338869 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.338881 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.339184 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.339216 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:10:42.340133 (+ 917us) client_negotiation.cc:240] Negotiation successful
0925 05:10:42.340261 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2547,"mutex_wait_us":78,"thread_start_us":2406,"threads_started":1}
I20260925 05:10:42.343631 24972 tablet_service.cc:1467] Processing CreateTablet for tablet 15d5a39ee22043d99e84ac00e35b5d64 (DEFAULT_TABLE table=test-table [id=0c54d815dfd546eb856da1ba127a902c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:42.344327 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 15d5a39ee22043d99e84ac00e35b5d64 (DEFAULT_TABLE table=test-table [id=0c54d815dfd546eb856da1ba127a902c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:42.344328 24704 tablet_service.cc:1467] Processing CreateTablet for tablet 15d5a39ee22043d99e84ac00e35b5d64 (DEFAULT_TABLE table=test-table [id=0c54d815dfd546eb856da1ba127a902c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:42.345271 24972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d5a39ee22043d99e84ac00e35b5d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:42.345693 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d5a39ee22043d99e84ac00e35b5d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:42.345698 24704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d5a39ee22043d99e84ac00e35b5d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:42.358240 25055 tablet_bootstrap.cc:492] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Bootstrap starting.
I20260925 05:10:42.360374 25057 tablet_bootstrap.cc:492] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Bootstrap starting.
I20260925 05:10:42.361158 25056 tablet_bootstrap.cc:492] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Bootstrap starting.
I20260925 05:10:42.361725 25055 tablet_bootstrap.cc:654] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:42.363047 25055 log.cc:824] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:42.364863 25056 tablet_bootstrap.cc:654] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:42.366237 25056 log.cc:824] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:42.368043 25055 tablet_bootstrap.cc:492] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: No bootstrap required, opened a new log
I20260925 05:10:42.368358 25055 ts_tablet_manager.cc:1397] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260925 05:10:42.368424 25057 tablet_bootstrap.cc:654] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:42.369712 25057 log.cc:824] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:42.370823 25056 tablet_bootstrap.cc:492] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: No bootstrap required, opened a new log
I20260925 05:10:42.371114 25056 ts_tablet_manager.cc:1397] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 05:10:42.372747 25057 tablet_bootstrap.cc:492] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: No bootstrap required, opened a new log
I20260925 05:10:42.372998 25057 ts_tablet_manager.cc:1397] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260925 05:10:42.376803 25055 raft_consensus.cc:348] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.377810 25055 raft_consensus.cc:374] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:42.377942 25055 raft_consensus.cc:729] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7571025ed4cd4a13abe5ca860a781e8d, State: Initialized, Role: FOLLOWER
I20260925 05:10:42.378582 25055 consensus_queue.cc:260] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.379655 25037 heartbeater.cc:503] Master 127.8.215.62:44205 was elected leader, sending a full tablet report...
I20260925 05:10:42.381891 25056 raft_consensus.cc:348] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.382340 25056 raft_consensus.cc:374] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:42.382494 25056 raft_consensus.cc:729] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bf15a821154e87b3509ae851ae77a7, State: Initialized, Role: FOLLOWER
I20260925 05:10:42.382946 25055 ts_tablet_manager.cc:1428] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260925 05:10:42.383265 25056 consensus_queue.cc:260] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.383621 25057 raft_consensus.cc:348] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.384183 25057 raft_consensus.cc:374] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:42.384308 25057 raft_consensus.cc:729] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed39eeb61eb74af3a53a9ab6e27b5fb2, State: Initialized, Role: FOLLOWER
I20260925 05:10:42.385064 25057 consensus_queue.cc:260] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.386180 25056 ts_tablet_manager.cc:1428] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260925 05:10:42.387637 24903 heartbeater.cc:503] Master 127.8.215.62:44205 was elected leader, sending a full tablet report...
I20260925 05:10:42.388351 25057 ts_tablet_manager.cc:1428] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260925 05:10:42.410459 25061 raft_consensus.cc:482] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:42.410787 25061 raft_consensus.cc:504] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.412551 25061 leader_election.cc:288] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2:44793), 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367)
I20260925 05:10:42.418292 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.413093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:50949 (local address 127.8.215.1:37367)
0925 05:10:42.413323 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.413333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.413415 (+ 82us) server_negotiation.cc:407] Connection header received
0925 05:10:42.413647 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.413652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.413713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.413811 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:42.414207 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.414952 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.416557 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.417402 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.417552 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.417657 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.417917 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.418044 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:10:42.418314 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.412987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37367 (local address 127.8.215.3:50949)
0925 05:10:42.413319 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.413340 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:42.413492 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:42.413832 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:42.413837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:42.413865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:42.414056 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.414065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.415106 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.415111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.416382 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.416394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.417569 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.417576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.417761 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.417785 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:10:42.417891 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 05:10:42.418036 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260925 05:10:42.419140 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.413018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:44793 (local address 127.8.215.3:52499)
0925 05:10:42.413880 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.413894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:42.414050 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:42.414420 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:42.414424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:42.414446 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:42.414774 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.414783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.415786 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.415792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.417214 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:42.417225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.418363 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.418369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:42.418589 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.418625 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:10:42.418738 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:10:42.418868 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":496,"threads_started":1}
I20260925 05:10:42.419204 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.413109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:52499 (local address 127.8.215.2:44793)
0925 05:10:42.413344 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.413351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.414001 (+ 650us) server_negotiation.cc:407] Connection header received
0925 05:10:42.414241 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.414245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.414305 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.414406 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:42.414933 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.415632 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.417374 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.418151 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.418311 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.418408 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.418780 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.418917 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 05:10:42.419536 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5a39ee22043d99e84ac00e35b5d64" candidate_uuid: "7571025ed4cd4a13abe5ca860a781e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60bf15a821154e87b3509ae851ae77a7" is_pre_election: true
I20260925 05:10:42.420145 24724 raft_consensus.cc:2454] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7571025ed4cd4a13abe5ca860a781e8d in term 0.
I20260925 05:10:42.420197 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5a39ee22043d99e84ac00e35b5d64" candidate_uuid: "7571025ed4cd4a13abe5ca860a781e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" is_pre_election: true
I20260925 05:10:42.420779 24858 raft_consensus.cc:2454] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7571025ed4cd4a13abe5ca860a781e8d in term 0.
I20260925 05:10:42.420841 24925 leader_election.cc:302] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bf15a821154e87b3509ae851ae77a7, 7571025ed4cd4a13abe5ca860a781e8d; no voters:
I20260925 05:10:42.421255 25061 raft_consensus.cc:2781] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:42.421401 25061 raft_consensus.cc:482] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:42.421535 25061 raft_consensus.cc:3037] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:42.423715 25061 raft_consensus.cc:504] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.424343 25061 leader_election.cc:288] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [CANDIDATE]: Term 1 election: Requested vote from peers ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2:44793), 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367)
I20260925 05:10:42.424790 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5a39ee22043d99e84ac00e35b5d64" candidate_uuid: "7571025ed4cd4a13abe5ca860a781e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2"
I20260925 05:10:42.424852 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5a39ee22043d99e84ac00e35b5d64" candidate_uuid: "7571025ed4cd4a13abe5ca860a781e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60bf15a821154e87b3509ae851ae77a7"
I20260925 05:10:42.424989 24858 raft_consensus.cc:3037] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:42.425067 24724 raft_consensus.cc:3037] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:42.427592 24724 raft_consensus.cc:2454] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7571025ed4cd4a13abe5ca860a781e8d in term 1.
I20260925 05:10:42.427608 24858 raft_consensus.cc:2454] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7571025ed4cd4a13abe5ca860a781e8d in term 1.
I20260925 05:10:42.427977 24925 leader_election.cc:302] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bf15a821154e87b3509ae851ae77a7, 7571025ed4cd4a13abe5ca860a781e8d; no voters:
I20260925 05:10:42.428261 25061 raft_consensus.cc:2781] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:42.428668 25061 raft_consensus.cc:686] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 1 LEADER]: Becoming Leader. State: Replica: 7571025ed4cd4a13abe5ca860a781e8d, State: Running, Role: LEADER
I20260925 05:10:42.429193 25061 consensus_queue.cc:237] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } }
I20260925 05:10:42.434017 24580 catalog_manager.cc:5495] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3). New cstate: current_term: 1 leader_uuid: "7571025ed4cd4a13abe5ca860a781e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:42.462432 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.456841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.8.215.3:45163)
0925 05:10:42.457073 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.457079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.457269 (+ 190us) server_negotiation.cc:407] Connection header received
0925 05:10:42.457508 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.457512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.457588 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.457748 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:42.458130 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.458866 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.460298 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.460713 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.460869 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.461353 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:42.461867 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:42.462041 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.462091 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.462170 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:10:42.470463 24724 raft_consensus.cc:1261] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Refusing update from remote peer 7571025ed4cd4a13abe5ca860a781e8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:42.470590 24858 raft_consensus.cc:1261] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7571025ed4cd4a13abe5ca860a781e8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:42.471511 25065 consensus_queue.cc:1035] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:42.471853 25061 consensus_queue.cc:1035] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:10:42.474895 24770 tablet.cc:2369] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:42.484432 25069 mvcc.cc:205] Tried to move back new op lower bound from 7333122221949145088 to 7333122221794033664. Current Snapshot: MvccSnapshot[applied={T|T < 7333122221949145088}]
I20260925 05:10:42.496693 24580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:42.497012 24580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:42.500802 24580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Sending DeleteTablet for 3 replicas of tablet 15d5a39ee22043d99e84ac00e35b5d64
I20260925 05:10:42.501847 24972 tablet_service.cc:1514] Processing DeleteTablet for tablet 15d5a39ee22043d99e84ac00e35b5d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:42 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:42892
I20260925 05:10:42.502096 24704 tablet_service.cc:1514] Processing DeleteTablet for tablet 15d5a39ee22043d99e84ac00e35b5d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:42 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:41094
I20260925 05:10:42.502794 24838 tablet_service.cc:1514] Processing DeleteTablet for tablet 15d5a39ee22043d99e84ac00e35b5d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:42 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:36584
I20260925 05:10:42.503019 25079 tablet_replica.cc:330] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: stopping tablet replica
I20260925 05:10:42.503484 25079 raft_consensus.cc:2229] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:42.503630 25080 tablet_replica.cc:330] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: stopping tablet replica
I20260925 05:10:42.503762 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24641
I20260925 05:10:42.504091 25079 raft_consensus.cc:2258] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:42.504081 25080 raft_consensus.cc:2229] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:42.504516 25080 raft_consensus.cc:2258] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:42.506430 25079 ts_tablet_manager.cc:1905] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:42.506913 25080 ts_tablet_manager.cc:1905] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:42.513975 25080 ts_tablet_manager.cc:1918] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:42.514276 25080 log.cc:1196] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/15d5a39ee22043d99e84ac00e35b5d64
I20260925 05:10:42.514876 25080 ts_tablet_manager.cc:1939] T 15d5a39ee22043d99e84ac00e35b5d64 P ed39eeb61eb74af3a53a9ab6e27b5fb2: Deleting consensus metadata
I20260925 05:10:42.516860 24566 catalog_manager.cc:4841] TS ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2:44793): tablet 15d5a39ee22043d99e84ac00e35b5d64 (table test-table [id=0c54d815dfd546eb856da1ba127a902c]) successfully deleted
I20260925 05:10:42.517154 25079 ts_tablet_manager.cc:1918] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:42.518635 25078 tablet_replica.cc:330] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: stopping tablet replica
I20260925 05:10:42.519111 25078 raft_consensus.cc:2229] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:42.519779 25078 raft_consensus.cc:2258] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:42.519922 25079 log.cc:1196] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/15d5a39ee22043d99e84ac00e35b5d64
I20260925 05:10:42.520817 25079 ts_tablet_manager.cc:1939] T 15d5a39ee22043d99e84ac00e35b5d64 P 60bf15a821154e87b3509ae851ae77a7: Deleting consensus metadata
I20260925 05:10:42.522037 25078 ts_tablet_manager.cc:1905] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:42.522719 24564 catalog_manager.cc:4841] TS 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367): tablet 15d5a39ee22043d99e84ac00e35b5d64 (table test-table [id=0c54d815dfd546eb856da1ba127a902c]) successfully deleted
Sep 25 05:10:42 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:10:42.529757 25078 ts_tablet_manager.cc:1918] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:42.529950 25078 log.cc:1196] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/15d5a39ee22043d99e84ac00e35b5d64
I20260925 05:10:42.530493 25078 ts_tablet_manager.cc:1939] T 15d5a39ee22043d99e84ac00e35b5d64 P 7571025ed4cd4a13abe5ca860a781e8d: Deleting consensus metadata
I20260925 05:10:42.531844 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.506127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.8.215.1:37367)
0925 05:10:42.506299 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.506305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.510206 (+ 3901us) server_negotiation.cc:407] Connection header received
0925 05:10:42.510509 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.510514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.510625 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.510713 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:42.517382 (+ 6669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.518196 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.519807 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.520559 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.520692 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.525731 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:42.525749 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:42.525763 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:42.525798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:42.528829 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.529559 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.529570 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.529578 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.529713 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.530138 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.530143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.530146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.530526 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:42.530855 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.531211 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.531435 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:10:42.532378 24566 catalog_manager.cc:4841] TS 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3:45163): tablet 15d5a39ee22043d99e84ac00e35b5d64 (table test-table [id=0c54d815dfd546eb856da1ba127a902c]) successfully deleted
I20260925 05:10:42.532965 24764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:42.645999 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24641
I20260925 05:10:42.670861 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24775
Sep 25 05:10:42 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:10:42.687057 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.671683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.8.215.2:44793)
0925 05:10:42.671891 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.671897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.671917 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:10:42.672057 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.672062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.672148 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.672252 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:42.673717 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.674452 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.675686 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.676077 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.676206 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.680859 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:42.680874 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:42.680890 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:42.680924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:42.683775 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.684563 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.684575 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.684578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.684712 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.685356 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.685362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.685366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.685802 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:42.686132 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.686430 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.686690 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 05:10:42.687682 24898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:42.777288 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24775
I20260925 05:10:42.795030 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24909
Sep 25 05:10:42 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:10:42.813841 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.795857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.8.215.3:45163)
0925 05:10:42.796058 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:42.796062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:42.796082 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:10:42.796321 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:42.796325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:42.796382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:42.796460 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:42.798267 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.799328 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.801342 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:42.801796 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:42.801890 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:42.807267 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:42.807282 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:42.807301 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:42.807344 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:42.810566 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.811415 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.811423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.811425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.811626 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:42.812010 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:42.812015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:42.812018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:42.812388 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:42.812714 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:42.813255 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 05:10:42.813483 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 05:10:42.814563 25032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:42.941653 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24909
I20260925 05:10:42.962215 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 24548
I20260925 05:10:42.963416 24610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:10:43.072150 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 24548
I20260925 05:11:03.095182 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44205
--webserver_interface=127.8.215.62
--webserver_port=40091
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:03.435223 25086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:03.435532 25086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:03.435683 25086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:03.443303 25086 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:11:03.443414 25086 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:11:03.443480 25086 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:11:03.443567 25086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:03.443670 25086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:03.443789 25086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:03.443879 25086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:03.443964 25086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:03.454214 25086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=40091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:03.455592 25086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:03.457069 25086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:03.481994 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:03.481940 25093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:03.481902 25095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:03.482885 25086 server_base.cc:1034] running on GCE node
I20260925 05:11:03.483880 25086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:03.485591 25086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:03.486856 25086 hybrid_clock.cc:648] HybridClock initialized: now 1790313063486772 us; error 57 us; skew 500 ppm
I20260925 05:11:03.487262 25086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:03 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313063, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:03.495649 25086 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:11:03.497478 25086 webserver.cc:460] Webserver started at http://127.8.215.62:40091/ using document root <none> and password file <none>
I20260925 05:11:03.498322 25086 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:03.498476 25086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:03.504680 25086 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 05:11:03.507743 25102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:03.509176 25086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:03.509373 25086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "2c5f9884285b41b3ae5f1ab53572566d"
format_stamp: "Formatted at 2026-09-25 05:10:40 on dist-test-slave-wpdm"
server_key: "3aca3280dcb72425c1954daf8cc6c3ea"
server_key_iv: "a34229233a0803b5d5bbf23427b571c3"
server_key_version: "encryptionkey@0"
I20260925 05:11:03.510464 25086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:03.528505 25086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:03.532953 25086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:03.533264 25086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:03.552893 25086 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:44205
I20260925 05:11:03.552917 25153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:44205 every 8 connection(s)
I20260925 05:11:03.554564 25086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:03.561817 25154 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:11:03.562296 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25086
I20260925 05:11:03.563308 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:37367
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:44205
--webserver_port=33867
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:11:03.567157 25154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap starting.
I20260925 05:11:03.589660 25154 log.cc:824] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:03.602413 25154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:11:03.603008 25154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap complete.
I20260925 05:11:03.612540 25154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:03.613457 25154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5f9884285b41b3ae5f1ab53572566d, State: Initialized, Role: FOLLOWER
I20260925 05:11:03.614293 25154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:03.614553 25154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:03.614681 25154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:03.614876 25154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:11:03.618999 25154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:03.619581 25154 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5f9884285b41b3ae5f1ab53572566d; no voters:
I20260925 05:11:03.620265 25154 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:11:03.620422 25158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:11:03.621724 25158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 LEADER]: Becoming Leader. State: Replica: 2c5f9884285b41b3ae5f1ab53572566d, State: Running, Role: LEADER
I20260925 05:11:03.622329 25158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:03.622850 25154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:03.625293 25159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c5f9884285b41b3ae5f1ab53572566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } } }
I20260925 05:11:03.625743 25159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:03.629521 25160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5f9884285b41b3ae5f1ab53572566d. Latest consensus state: current_term: 2 leader_uuid: "2c5f9884285b41b3ae5f1ab53572566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } } }
I20260925 05:11:03.629899 25160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:03.630383 25164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:03.640018 25164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:03.642464 25164 catalog_manager.cc:1259] Loaded cluster ID: 6cb57907b32f48c39769df240b3315e6
I20260925 05:11:03.642685 25164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:03.648595 25164 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:03.651101 25164 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Loaded TSK: 0
I20260925 05:11:03.690419 25175 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:11:03.692495 25164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Generated new TSK 1
I20260925 05:11:03.693006 25164 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:11:03.928447 25156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:03.928786 25156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:03.928965 25156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:03.936502 25156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:03.936658 25156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:03.936862 25156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:11:03.946678 25156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:37367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=33867
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:03.948009 25156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:03.949455 25156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:03.964481 25181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:03.964524 25182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:03.964708 25184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:03.970530 25156 server_base.cc:1034] running on GCE node
I20260925 05:11:03.971292 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:03.972395 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:03.973630 25156 hybrid_clock.cc:648] HybridClock initialized: now 1790313063973579 us; error 46 us; skew 500 ppm
I20260925 05:11:03.973978 25156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:03 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313063, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:03.977375 25156 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:11:03.979159 25156 webserver.cc:460] Webserver started at http://127.8.215.1:33867/ using document root <none> and password file <none>
I20260925 05:11:03.979909 25156 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:03.980113 25156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:03.985788 25156 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:11:03.988871 25191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:03.990343 25156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:11:03.990509 25156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "60bf15a821154e87b3509ae851ae77a7"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "9600098b6d49da5c1f6eb1275309dc23"
server_key_iv: "5d4f6a25c2c19280e9cef23d83d7fee5"
server_key_version: "encryptionkey@0"
I20260925 05:11:03.991576 25156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:04.020573 25156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:04.025169 25156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:04.025437 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:04.026582 25156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:04.027925 25156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:04.028010 25156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.028167 25156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:04.028282 25156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.062026 25156 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:37367
I20260925 05:11:04.062194 25303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:37367 every 8 connection(s)
I20260925 05:11:04.063500 25156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:04.072757 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25156
I20260925 05:11:04.073567 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:44793
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:44205
--webserver_port=39995
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:11:04 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313063, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:04.090533 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.066523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:47131 (local address 127.8.215.62:44205)
0925 05:11:04.067187 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:04.067203 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:04.067950 (+ 747us) server_negotiation.cc:407] Connection header received
0925 05:11:04.071356 (+ 3406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:04.071377 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:04.071961 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:04.072449 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:04.074314 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.076339 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.077993 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.078674 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.078888 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.082514 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:04.082522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:04.082536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:04.082574 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:04.085656 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:04.086440 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:04.086447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:04.086453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:04.086616 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:04.087080 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:04.087084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:04.087086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:04.087662 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:04.087870 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:04.088701 (+ 831us) server_negotiation.cc:299] Negotiation successful
0925 05:11:04.089147 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":255,"threads_started":1}
I20260925 05:11:04.090512 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.066821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:47131)
0925 05:11:04.067751 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:04.067803 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:04.070510 (+ 2707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:04.072602 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:04.072614 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:04.073334 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:04.074025 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.074086 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.076510 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.076516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.077825 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.077837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.078930 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.078947 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.079232 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.079916 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:04.079946 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:04.082199 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:04.085812 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:04.085822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:04.085838 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:04.086280 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:04.086737 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:04.086742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:04.086746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:04.086951 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:04.087814 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:04.087822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:04.088175 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 05:11:04.088457 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 05:11:04.089143 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":419,"threads_started":1}
I20260925 05:11:04.094399 25304 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44205
I20260925 05:11:04.094739 25304 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:04.095548 25304 heartbeater.cc:511] Master 127.8.215.62:44205 requested a full tablet report, sending...
I20260925 05:11:04.097846 25119 ts_manager.cc:194] Registered new tserver with Master: 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367)
I20260925 05:11:04.099380 25119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:47131
W20260925 05:11:04.405644 25308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:04.406018 25308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:04.406196 25308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:04.413569 25308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:04.413723 25308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:04.413898 25308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:11:04.423892 25308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:44793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=39995
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:04.425232 25308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:04.426585 25308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:04.440397 25314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.440575 25316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.441068 25313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:04.441484 25308 server_base.cc:1034] running on GCE node
I20260925 05:11:04.442155 25308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:04.443131 25308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:04.444391 25308 hybrid_clock.cc:648] HybridClock initialized: now 1790313064444323 us; error 58 us; skew 500 ppm
I20260925 05:11:04.444824 25308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:04 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313064, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:04.448172 25308 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:11:04.449765 25308 webserver.cc:460] Webserver started at http://127.8.215.2:39995/ using document root <none> and password file <none>
I20260925 05:11:04.450781 25308 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:04.450904 25308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:04.456414 25308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:11:04.459164 25323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.460481 25308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:11:04.460609 25308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "0e59a88f933536b08a9bb62ea7258632"
server_key_iv: "333ebbb9c3dfe7cbc661f165bad02edf"
server_key_version: "encryptionkey@0"
I20260925 05:11:04.461634 25308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:04.486884 25308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:04.491783 25308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:04.492035 25308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:04.493053 25308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:04.494359 25308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:04.494446 25308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.494527 25308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:04.494583 25308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.527804 25308 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:44793
I20260925 05:11:04.527927 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:44793 every 8 connection(s)
I20260925 05:11:04.529326 25308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:04.539446 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25308
I20260925 05:11:04.540179 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:45163
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:44205
--webserver_port=37201
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:11:04 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313064, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:04.550362 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.531942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:37519 (local address 127.8.215.62:44205)
0925 05:11:04.532154 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:04.532162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:04.533389 (+ 1227us) server_negotiation.cc:407] Connection header received
0925 05:11:04.535163 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:04.535168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:04.535239 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:04.535327 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:04.536814 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.537494 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.539218 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.539708 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.539806 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.544295 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:04.544299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:04.544302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:04.544336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:04.546889 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:04.547838 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:04.547842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:04.547845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:04.547963 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:04.548445 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:04.548448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:04.548450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:04.548795 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:04.548969 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:04.549695 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 05:11:04.550017 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:11:04.551363 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.532496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:37519)
0925 05:11:04.533219 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:04.533262 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:04.534897 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:04.535534 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:04.535542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:04.536072 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:04.536624 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.536640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.537686 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.537691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.539064 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.539083 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.539940 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.539947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.540266 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.541040 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:04.541084 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:04.544039 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:04.547037 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:04.547058 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:04.547079 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:04.547719 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:04.548077 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:04.548084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:04.548090 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:04.548331 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:04.548961 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:04.548967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:04.549304 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 05:11:04.549542 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 05:11:04.550025 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":302,"threads_started":1}
I20260925 05:11:04.553020 25436 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44205
I20260925 05:11:04.553339 25436 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:04.554203 25436 heartbeater.cc:511] Master 127.8.215.62:44205 requested a full tablet report, sending...
I20260925 05:11:04.555892 25119 ts_manager.cc:194] Registered new tserver with Master: ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2:44793)
I20260925 05:11:04.557291 25119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:37519
W20260925 05:11:04.876295 25439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:04.876617 25439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:04.876828 25439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:04.884521 25439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:04.884642 25439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:04.884863 25439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:11:04.894836 25439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:45163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=37201
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:04.896204 25439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:04.897634 25439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:04.910643 25447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.910674 25445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.910843 25444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:04.911365 25439 server_base.cc:1034] running on GCE node
I20260925 05:11:04.911947 25439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:04.912922 25439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:04.914143 25439 hybrid_clock.cc:648] HybridClock initialized: now 1790313064914103 us; error 59 us; skew 500 ppm
I20260925 05:11:04.914443 25439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:04 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313064, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:04.917550 25439 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:11:04.918983 25439 webserver.cc:460] Webserver started at http://127.8.215.3:37201/ using document root <none> and password file <none>
I20260925 05:11:04.919672 25439 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:04.919827 25439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:04.925179 25439 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:04.927767 25454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.929308 25439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:04.929481 25439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "7571025ed4cd4a13abe5ca860a781e8d"
format_stamp: "Formatted at 2026-09-25 05:10:42 on dist-test-slave-wpdm"
server_key: "368818a203fbd8ef2910007a2ed90aa7"
server_key_iv: "645c4694093dd0c04bbf2cec0b4ff9de"
server_key_version: "encryptionkey@0"
I20260925 05:11:04.930469 25439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:04.950914 25439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:04.955120 25439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:04.955389 25439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:04.956315 25439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:04.957629 25439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:04.957747 25439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.957870 25439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:04.957965 25439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.991022 25439 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:45163
I20260925 05:11:04.991086 25566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:45163 every 8 connection(s)
I20260925 05:11:04.992544 25439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:04.999779 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25439
Sep 25 05:11:05 dist-test-slave-wpdm krb5kdc[24535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313064, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:05.012972 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.995413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:49991 (local address 127.8.215.62:44205)
0925 05:11:04.995614 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:04.995624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:04.996923 (+ 1299us) server_negotiation.cc:407] Connection header received
0925 05:11:04.998751 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:04.998757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:04.998843 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:04.998961 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.000507 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.001522 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.003033 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.003470 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.003618 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.007097 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.007103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.007108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:05.007173 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.009964 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.010684 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.010689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.010692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.010827 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.011250 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.011255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.011258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.011526 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.011695 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.012362 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.012579 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:11:05.013959 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.995949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:49991)
0925 05:11:04.996746 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:04.996799 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:04.998476 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:04.999210 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:04.999218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:04.999714 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.000334 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.000352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.001690 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.001696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.002890 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.002901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.003701 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.003711 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.003940 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.004627 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:05.004648 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.006890 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.010117 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:05.010126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:05.010143 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:05.010552 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:05.010939 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:05.010944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:05.010948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:05.011137 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:05.011665 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.011672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.012010 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.012288 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.012853 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":271,"threads_started":1}
I20260925 05:11:05.015167 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.000669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.8.215.62:44205)
0925 05:11:05.001253 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.001262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.001826 (+ 564us) server_negotiation.cc:407] Connection header received
0925 05:11:05.002012 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.002017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.002097 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.002207 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.003824 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.004577 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.006014 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.006540 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.006672 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.009517 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.009522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.009525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:05.009565 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.012158 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.012872 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.012877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.012882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.012995 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.013509 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.013514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.013517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.013779 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.014038 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.014410 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.014619 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260925 05:11:05.015501 25567 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44205
I20260925 05:11:05.015861 25567 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.016780 25567 heartbeater.cc:511] Master 127.8.215.62:44205 requested a full tablet report, sending...
I20260925 05:11:05.018179 25119 ts_manager.cc:194] Registered new tserver with Master: 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3:45163)
I20260925 05:11:05.018949 25119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:49991
I20260925 05:11:05.029913 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:05.036571 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.030847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.8.215.62:44205)
0925 05:11:05.031068 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.031074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.031416 (+ 342us) server_negotiation.cc:407] Connection header received
0925 05:11:05.031609 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.031613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.031742 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.031859 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:05.032247 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.033003 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.034539 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.035171 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.035268 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.035828 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:05.036024 (+ 196us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 05:11:05.036304 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33896: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 05:11:05.036703 25570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33896: token expired
W20260925 05:11:05.037277 25043 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.215.62:44205)
I20260925 05:11:05.049594 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.037743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.8.215.62:44205)
0925 05:11:05.037892 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.037896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.037914 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:11:05.038047 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.038051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.038109 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.038182 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.039521 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.040223 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.041619 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.041999 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.042088 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.044716 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.044721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.044770 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:05.044812 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.047174 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.047822 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.047827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.047829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.047950 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.048286 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.048289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.048291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.048590 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.048846 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.049089 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.049247 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:11:05.053048 25119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:05.060266 25119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:05.083487 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.073079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.8.215.1:37367)
0925 05:11:05.074939 (+ 1860us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.074948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.075658 (+ 710us) server_negotiation.cc:407] Connection header received
0925 05:11:05.075767 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.075778 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.076038 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.076238 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:05.076816 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.078322 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.081125 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.082297 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.082449 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.082686 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.082955 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.083114 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":2346,"threads_started":1}
I20260925 05:11:05.083719 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.072712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.8.215.2:44793)
0925 05:11:05.073311 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.073319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.074059 (+ 740us) server_negotiation.cc:407] Connection header received
0925 05:11:05.074494 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.074503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.074804 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.074980 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:05.076386 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.078026 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.081152 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.082354 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.082482 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.082680 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.083193 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.083368 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":335,"threads_started":1}
I20260925 05:11:05.084301 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.072924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37367 (local address 127.0.0.1:37534)
0925 05:11:05.075373 (+ 2449us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.075389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.075514 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.076237 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.076245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.076272 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:05.076618 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.076629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.078469 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.078477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.080935 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.080950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.082510 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.082516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.082714 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.082770 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.082909 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.083114 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2318,"mutex_wait_us":25,"thread_start_us":2169,"threads_started":1}
I20260925 05:11:05.084748 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.072521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:44793 (local address 127.0.0.1:37634)
0925 05:11:05.073900 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.073946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.074230 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.075331 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.075339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.075380 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:05.076206 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.076218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.078172 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.078177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.080813 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.080948 (+ 135us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.082815 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.082822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.082981 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.083014 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.083810 (+ 796us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.084004 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":239,"thread_start_us":963,"threads_started":1}
I20260925 05:11:05.084991 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.075168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.8.215.3:45163)
0925 05:11:05.076161 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.076169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.076200 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:11:05.076287 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.076296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.076519 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.076709 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:05.077631 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.079198 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.081775 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.083115 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.083227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.083447 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.084128 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.084273 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":684,"threads_started":1}
I20260925 05:11:05.085232 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.074517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:45163 (local address 127.0.0.1:44672)
0925 05:11:05.075733 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.075753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.076027 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.076994 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.076998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.077020 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:05.077428 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.077441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.079342 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.079347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.081610 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.081624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.083722 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.083731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.083910 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.083955 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.084073 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.084201 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":3,"thread_start_us":1084,"threads_started":1}
I20260925 05:11:05.088239 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 4dda391ba6f94efeba530afedc2221bd (DEFAULT_TABLE table=test-table [id=c29f8af4685641e4b169b9b9ad8a8314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:05.088963 25371 tablet_service.cc:1467] Processing CreateTablet for tablet 4dda391ba6f94efeba530afedc2221bd (DEFAULT_TABLE table=test-table [id=c29f8af4685641e4b169b9b9ad8a8314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:05.088967 25502 tablet_service.cc:1467] Processing CreateTablet for tablet 4dda391ba6f94efeba530afedc2221bd (DEFAULT_TABLE table=test-table [id=c29f8af4685641e4b169b9b9ad8a8314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:05.089799 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dda391ba6f94efeba530afedc2221bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:05.090253 25502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dda391ba6f94efeba530afedc2221bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:05.090253 25371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dda391ba6f94efeba530afedc2221bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:05.102007 25583 tablet_bootstrap.cc:492] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Bootstrap starting.
I20260925 05:11:05.102682 25304 heartbeater.cc:503] Master 127.8.215.62:44205 was elected leader, sending a full tablet report...
I20260925 05:11:05.106455 25583 tablet_bootstrap.cc:654] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:05.107981 25583 log.cc:824] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:05.110225 25585 tablet_bootstrap.cc:492] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Bootstrap starting.
I20260925 05:11:05.111021 25583 tablet_bootstrap.cc:492] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: No bootstrap required, opened a new log
I20260925 05:11:05.111305 25583 ts_tablet_manager.cc:1397] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260925 05:11:05.114372 25584 tablet_bootstrap.cc:492] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Bootstrap starting.
I20260925 05:11:05.114889 25585 tablet_bootstrap.cc:654] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:05.116276 25585 log.cc:824] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:05.118129 25584 tablet_bootstrap.cc:654] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:05.119091 25585 tablet_bootstrap.cc:492] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: No bootstrap required, opened a new log
I20260925 05:11:05.119385 25585 ts_tablet_manager.cc:1397] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 05:11:05.119374 25584 log.cc:824] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:05.121142 25583 raft_consensus.cc:348] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.121590 25583 raft_consensus.cc:374] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:05.121701 25583 raft_consensus.cc:729] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bf15a821154e87b3509ae851ae77a7, State: Initialized, Role: FOLLOWER
I20260925 05:11:05.122581 25584 tablet_bootstrap.cc:492] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: No bootstrap required, opened a new log
I20260925 05:11:05.122419 25583 consensus_queue.cc:260] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.122892 25584 ts_tablet_manager.cc:1397] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:11:05.124351 25583 ts_tablet_manager.cc:1428] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 05:11:05.129336 25585 raft_consensus.cc:348] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.129693 25585 raft_consensus.cc:374] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:05.129820 25585 raft_consensus.cc:729] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7571025ed4cd4a13abe5ca860a781e8d, State: Initialized, Role: FOLLOWER
I20260925 05:11:05.130414 25585 consensus_queue.cc:260] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.130921 25584 raft_consensus.cc:348] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.131366 25584 raft_consensus.cc:374] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:05.131479 25584 raft_consensus.cc:729] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed39eeb61eb74af3a53a9ab6e27b5fb2, State: Initialized, Role: FOLLOWER
I20260925 05:11:05.131592 25567 heartbeater.cc:503] Master 127.8.215.62:44205 was elected leader, sending a full tablet report...
I20260925 05:11:05.132074 25585 ts_tablet_manager.cc:1428] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260925 05:11:05.132046 25584 consensus_queue.cc:260] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.133241 25436 heartbeater.cc:503] Master 127.8.215.62:44205 was elected leader, sending a full tablet report...
I20260925 05:11:05.133911 25584 ts_tablet_manager.cc:1428] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 05:11:05.182567 25591 raft_consensus.cc:482] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:05.182857 25591 raft_consensus.cc:504] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.184487 25591 leader_election.cc:288] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367), 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3:45163)
I20260925 05:11:05.191846 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.185007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:42305 (local address 127.8.215.3:45163)
0925 05:11:05.185220 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.185227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.186148 (+ 921us) server_negotiation.cc:407] Connection header received
0925 05:11:05.186306 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.186313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.186395 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.186490 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:05.187163 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.188002 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.190111 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.190878 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.191031 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.191140 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.191473 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.191594 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:11:05.191843 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.185084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:49779 (local address 127.8.215.1:37367)
0925 05:11:05.185297 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.185305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.186307 (+ 1002us) server_negotiation.cc:407] Connection header received
0925 05:11:05.186448 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.186476 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.186548 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.186642 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:05.187293 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.188180 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.190085 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.190863 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.190998 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.191128 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.191434 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.191568 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:11:05.191998 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.185138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:37367 (local address 127.8.215.2:49779)
0925 05:11:05.186198 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.186216 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.186332 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.186691 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.186694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.186722 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:05.187144 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.187154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.188318 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.188321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.189906 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.189918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.191102 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.191109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.191271 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.191296 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.191371 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.191522 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":1,"thread_start_us":502,"threads_started":1}
I20260925 05:11:05.192022 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.185099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:45163 (local address 127.8.215.2:42305)
0925 05:11:05.186008 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.186023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.186160 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.186518 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.186522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.186554 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:05.187003 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.187016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.188129 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.188134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.189921 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.189933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.191135 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.191144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.191288 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.191311 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.191422 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.191570 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":24,"thread_start_us":604,"threads_started":1}
I20260925 05:11:05.193234 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dda391ba6f94efeba530afedc2221bd" candidate_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60bf15a821154e87b3509ae851ae77a7" is_pre_election: true
I20260925 05:11:05.193390 25522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dda391ba6f94efeba530afedc2221bd" candidate_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7571025ed4cd4a13abe5ca860a781e8d" is_pre_election: true
I20260925 05:11:05.193810 25259 raft_consensus.cc:2454] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed39eeb61eb74af3a53a9ab6e27b5fb2 in term 0.
I20260925 05:11:05.193982 25522 raft_consensus.cc:2454] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed39eeb61eb74af3a53a9ab6e27b5fb2 in term 0.
I20260925 05:11:05.194535 25324 leader_election.cc:302] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bf15a821154e87b3509ae851ae77a7, ed39eeb61eb74af3a53a9ab6e27b5fb2; no voters:
I20260925 05:11:05.194998 25591 raft_consensus.cc:2781] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:05.195158 25591 raft_consensus.cc:482] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:05.195261 25591 raft_consensus.cc:3037] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:05.197603 25591 raft_consensus.cc:504] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.198213 25591 leader_election.cc:288] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367), 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3:45163)
I20260925 05:11:05.198699 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dda391ba6f94efeba530afedc2221bd" candidate_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60bf15a821154e87b3509ae851ae77a7"
I20260925 05:11:05.198761 25522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dda391ba6f94efeba530afedc2221bd" candidate_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7571025ed4cd4a13abe5ca860a781e8d"
I20260925 05:11:05.198941 25522 raft_consensus.cc:3037] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:05.198930 25259 raft_consensus.cc:3037] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:05.202198 25522 raft_consensus.cc:2454] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed39eeb61eb74af3a53a9ab6e27b5fb2 in term 1.
I20260925 05:11:05.202198 25259 raft_consensus.cc:2454] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed39eeb61eb74af3a53a9ab6e27b5fb2 in term 1.
I20260925 05:11:05.202617 25324 leader_election.cc:302] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bf15a821154e87b3509ae851ae77a7, ed39eeb61eb74af3a53a9ab6e27b5fb2; no voters:
I20260925 05:11:05.202951 25591 raft_consensus.cc:2781] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:05.203416 25591 raft_consensus.cc:686] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 LEADER]: Becoming Leader. State: Replica: ed39eeb61eb74af3a53a9ab6e27b5fb2, State: Running, Role: LEADER
I20260925 05:11:05.203879 25591 consensus_queue.cc:237] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } }
I20260925 05:11:05.208546 25119 catalog_manager.cc:5495] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 44793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 } health_report { overall_health: UNKNOWN } } }
I20260925 05:11:05.243234 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.238257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.8.215.2:44793)
0925 05:11:05.238427 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.238431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.238449 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:11:05.238639 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.238643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.238721 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.238859 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:05.239183 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.239885 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.241295 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.241690 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.241783 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.242246 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:05.242728 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:05.242839 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.242940 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.243006 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 05:11:05.244403 25568 tablet.cc:2369] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:05.251876 25522 raft_consensus.cc:1261] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Refusing update from remote peer ed39eeb61eb74af3a53a9ab6e27b5fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:05.251904 25259 raft_consensus.cc:1261] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Refusing update from remote peer ed39eeb61eb74af3a53a9ab6e27b5fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:05.252859 25594 consensus_queue.cc:1035] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7571025ed4cd4a13abe5ca860a781e8d" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:05.253194 25591 consensus_queue.cc:1035] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60bf15a821154e87b3509ae851ae77a7" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:05.264408 25597 mvcc.cc:205] Tried to move back new op lower bound from 7333122315261218816 to 7333122315079032832. Current Snapshot: MvccSnapshot[applied={T|T < 7333122315261218816}]
I20260925 05:11:05.279438 25119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:05.279675 25119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:05.283223 25119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Sending DeleteTablet for 3 replicas of tablet 4dda391ba6f94efeba530afedc2221bd
I20260925 05:11:05.284371 25239 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dda391ba6f94efeba530afedc2221bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:05 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:37534
I20260925 05:11:05.284402 25502 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dda391ba6f94efeba530afedc2221bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:05 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:44672
I20260925 05:11:05.284649 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25156
I20260925 05:11:05.285388 25371 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dda391ba6f94efeba530afedc2221bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:05 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:37634
I20260925 05:11:05.285611 25606 tablet_replica.cc:330] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: stopping tablet replica
I20260925 05:11:05.286024 25606 raft_consensus.cc:2229] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:05.286566 25606 raft_consensus.cc:2258] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:05.287395 25607 tablet_replica.cc:330] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: stopping tablet replica
I20260925 05:11:05.287864 25607 raft_consensus.cc:2229] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:05.288364 25607 raft_consensus.cc:2258] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:05.291262 25607 ts_tablet_manager.cc:1905] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:05.295503 25606 ts_tablet_manager.cc:1905] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:05.298482 25607 ts_tablet_manager.cc:1918] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:05.298682 25607 log.cc:1196] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/4dda391ba6f94efeba530afedc2221bd
I20260925 05:11:05.299290 25607 ts_tablet_manager.cc:1939] T 4dda391ba6f94efeba530afedc2221bd P 60bf15a821154e87b3509ae851ae77a7: Deleting consensus metadata
I20260925 05:11:05.300000 25608 tablet_replica.cc:330] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: stopping tablet replica
I20260925 05:11:05.300478 25608 raft_consensus.cc:2229] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:05.301323 25103 catalog_manager.cc:4841] TS 60bf15a821154e87b3509ae851ae77a7 (127.8.215.1:37367): tablet 4dda391ba6f94efeba530afedc2221bd (table test-table [id=c29f8af4685641e4b169b9b9ad8a8314]) successfully deleted
I20260925 05:11:05.302218 25608 raft_consensus.cc:2258] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:05.303938 25606 ts_tablet_manager.cc:1918] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:05.304198 25606 log.cc:1196] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/4dda391ba6f94efeba530afedc2221bd
I20260925 05:11:05.304749 25608 ts_tablet_manager.cc:1905] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:05.304806 25606 ts_tablet_manager.cc:1939] T 4dda391ba6f94efeba530afedc2221bd P 7571025ed4cd4a13abe5ca860a781e8d: Deleting consensus metadata
I20260925 05:11:05.305295 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.286279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.8.215.1:37367)
0925 05:11:05.286450 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.286456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.286473 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:11:05.286527 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.286533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.286653 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.286751 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.288931 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.290081 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.292383 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.293070 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.293217 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.297189 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.297203 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.297221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:05.297270 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.301631 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.302640 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.302649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.302652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.302797 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.303260 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.303266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.303270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.303727 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.304002 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.304639 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.304894 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:11:05.306737 25299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:05.307248 25105 catalog_manager.cc:4841] TS 7571025ed4cd4a13abe5ca860a781e8d (127.8.215.3:45163): tablet 4dda391ba6f94efeba530afedc2221bd (table test-table [id=c29f8af4685641e4b169b9b9ad8a8314]) successfully deleted
I20260925 05:11:05.313004 25608 ts_tablet_manager.cc:1918] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:05.313206 25608 log.cc:1196] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/4dda391ba6f94efeba530afedc2221bd
I20260925 05:11:05.313745 25608 ts_tablet_manager.cc:1939] T 4dda391ba6f94efeba530afedc2221bd P ed39eeb61eb74af3a53a9ab6e27b5fb2: Deleting consensus metadata
I20260925 05:11:05.315600 25105 catalog_manager.cc:4841] TS ed39eeb61eb74af3a53a9ab6e27b5fb2 (127.8.215.2:44793): tablet 4dda391ba6f94efeba530afedc2221bd (table test-table [id=c29f8af4685641e4b169b9b9ad8a8314]) successfully deleted
I20260925 05:11:05.399819 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25156
I20260925 05:11:05.423076 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25308
I20260925 05:11:05.439003 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.423941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.8.215.2:44793)
0925 05:11:05.424133 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.424139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.424162 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:11:05.424334 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.424340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.424423 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.424542 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.425870 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.427164 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.428476 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.429226 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.429358 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.431917 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.431935 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.431955 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:05.432005 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.436156 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.436867 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.436877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.436881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.437058 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.437395 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.437400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.437404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.437870 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.438201 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.438356 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.438583 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:11:05.439762 25431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:05.558579 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25308
I20260925 05:11:05.580016 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25439
I20260925 05:11:05.595458 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.580885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.8.215.3:45163)
0925 05:11:05.581075 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.581080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.581102 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:11:05.581274 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.581278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.581362 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.581465 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.582902 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.583751 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.585590 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.586083 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.586200 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.589265 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.589280 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.589296 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:05.589333 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.592075 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.592944 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.592956 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.592961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.593180 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:05.593758 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:05.593766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:05.593772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:05.594345 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.594585 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.594863 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.595081 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:11:05.596215 25562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:05.685547 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25439
I20260925 05:11:05.705534 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25086
I20260925 05:11:05.707144 25149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:05.821471 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25086
I20260925 05:11:25.844316 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44205
--webserver_interface=127.8.215.62
--webserver_port=40091
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:26.201010 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:26.201553 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:26.201812 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:26.216141 25613 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:11:26.216305 25613 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:11:26.216400 25613 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:11:26.216502 25613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:26.216606 25613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:26.216742 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:26.216835 25613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:26.216933 25613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:26.232002 25613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=40091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:26.233561 25613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:26.235092 25613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:26.247180 25618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:26.247469 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:26.247820 25621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:26.248350 25613 server_base.cc:1034] running on GCE node
I20260925 05:11:26.249291 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:26.250734 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:26.251976 25613 hybrid_clock.cc:648] HybridClock initialized: now 1790313086251909 us; error 33 us; skew 500 ppm
I20260925 05:11:26.252332 25613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:26 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:26.255712 25613 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:11:26.257421 25613 webserver.cc:460] Webserver started at http://127.8.215.62:40091/ using document root <none> and password file <none>
I20260925 05:11:26.258173 25613 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:26.258306 25613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:26.263970 25613 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:11:26.266932 25628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:26.268342 25613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:11:26.268550 25613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "2c5f9884285b41b3ae5f1ab53572566d"
format_stamp: "Formatted at 2026-09-25 05:10:40 on dist-test-slave-wpdm"
server_key: "3aca3280dcb72425c1954daf8cc6c3ea"
server_key_iv: "a34229233a0803b5d5bbf23427b571c3"
server_key_version: "encryptionkey@0"
I20260925 05:11:26.269703 25613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:26.295478 25613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:26.300076 25613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:26.300364 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:26.319409 25613 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:44205
I20260925 05:11:26.319468 25679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:44205 every 8 connection(s)
I20260925 05:11:26.320982 25613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:26.327876 25680 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:11:26.330884 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25613
I20260925 05:11:26.331568 25680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap starting.
I20260925 05:11:26.331902 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:37367
--local_ip_for_outbound_sockets=127.8.215.1
--tserver_master_addrs=127.8.215.62:44205
--webserver_port=33867
--webserver_interface=127.8.215.1
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:11:26.355466 25680 log.cc:824] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:26.372423 25680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:11:26.373112 25680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Bootstrap complete.
I20260925 05:11:26.382664 25680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:26.383693 25680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5f9884285b41b3ae5f1ab53572566d, State: Initialized, Role: FOLLOWER
I20260925 05:11:26.384385 25680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:26.384635 25680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:26.384797 25680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:26.385004 25680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:11:26.389161 25680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:26.389637 25680 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5f9884285b41b3ae5f1ab53572566d; no voters:
I20260925 05:11:26.390347 25680 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:11:26.390483 25684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:11:26.391687 25684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [term 3 LEADER]: Becoming Leader. State: Replica: 2c5f9884285b41b3ae5f1ab53572566d, State: Running, Role: LEADER
I20260925 05:11:26.392280 25684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } }
I20260925 05:11:26.392596 25680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:26.394901 25686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5f9884285b41b3ae5f1ab53572566d. Latest consensus state: current_term: 3 leader_uuid: "2c5f9884285b41b3ae5f1ab53572566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } } }
I20260925 05:11:26.395263 25686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:26.395103 25685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2c5f9884285b41b3ae5f1ab53572566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5f9884285b41b3ae5f1ab53572566d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44205 } } }
I20260925 05:11:26.395526 25685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:26.405085 25691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:26.417510 25691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:26.419523 25691 catalog_manager.cc:1259] Loaded cluster ID: 6cb57907b32f48c39769df240b3315e6
I20260925 05:11:26.419723 25691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:26.425230 25691 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:26.427429 25691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Loaded TSK: 0
I20260925 05:11:26.427665 25691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c5f9884285b41b3ae5f1ab53572566d: Loaded TSK: 1
F20260925 05:11:26.429179 25691 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbd411dc9ad google::LogMessage::Fail() at ??:0
@ 0x7fbd411df88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbd411dc4de google::LogMessage::Flush() at ??:0
@ 0x7fbd411dda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbd44f407d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbd44f2f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbd56c0b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbd56c0a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbd56c962a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fbd56c960c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fbd56ce05e9 std::function<>::operator()() at ??:0
@ 0x7fbd56c0ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbd56c0cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbd56c94610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbd56c94439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbd43e28345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbd43e2b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbd43e2b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbd43e06eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbd470e06db start_thread at ??:0
@ 0x7fbd4007661f clone at ??:0
W20260925 05:11:26.726681 25682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:26.726971 25682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:26.727080 25682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:26.734724 25682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:26.734874 25682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:26.735015 25682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:11:26.745579 25682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:37367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=33867
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:26.746871 25682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:26.748152 25682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:26.761495 25730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:26.761489 25728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:26.761544 25727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:26.762573 25682 server_base.cc:1034] running on GCE node
I20260925 05:11:26.763235 25682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:26.764222 25682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:26.765470 25682 hybrid_clock.cc:648] HybridClock initialized: now 1790313086765410 us; error 51 us; skew 500 ppm
I20260925 05:11:26.765820 25682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:26 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:26.769073 25682 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:11:26.770596 25682 webserver.cc:460] Webserver started at http://127.8.215.1:33867/ using document root <none> and password file <none>
I20260925 05:11:26.771273 25682 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:26.771376 25682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:26.776981 25682 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:11:26.779753 25737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:26.781241 25682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:26.781391 25682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "60bf15a821154e87b3509ae851ae77a7"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "9600098b6d49da5c1f6eb1275309dc23"
server_key_iv: "5d4f6a25c2c19280e9cef23d83d7fee5"
server_key_version: "encryptionkey@0"
I20260925 05:11:26.782361 25682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:26.827941 25682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:26.832573 25682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:26.832919 25682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:26.834009 25682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:26.835453 25682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:26.835541 25682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:26.835618 25682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:26.835675 25682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:26.871209 25682 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:37367
I20260925 05:11:26.871322 25849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:37367 every 8 connection(s)
I20260925 05:11:26.872833 25682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:26.877750 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:26.875830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:45667)
0925 05:11:26.876580 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:26.876885 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":304,"threads_started":1}
W20260925 05:11:26.878535 25850 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:44205 (0 consecutive failures): Network error: Failed to ping master at 127.8.215.62:44205: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
I20260925 05:11:26.879380 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:26.879068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:56885)
0925 05:11:26.879205 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:26.879271 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 05:11:26.880148 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:26.879865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:37265)
0925 05:11:26.879982 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:26.880049 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:11:26.880409 25850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:11:26.881974 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25682
I20260925 05:11:26.882604 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:44793
--local_ip_for_outbound_sockets=127.8.215.2
--tserver_master_addrs=127.8.215.62:44205
--webserver_port=39995
--webserver_interface=127.8.215.2
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:27.237627 25853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:27.237943 25853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:27.238142 25853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:27.245606 25853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:27.245745 25853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:27.245967 25853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:11:27.256119 25853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:44793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=39995
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:27.257529 25853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:27.258939 25853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:27.273171 25858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:27.273218 25859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:27.273438 25861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:27.274333 25853 server_base.cc:1034] running on GCE node
I20260925 05:11:27.275027 25853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:27.276046 25853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:27.277272 25853 hybrid_clock.cc:648] HybridClock initialized: now 1790313087277211 us; error 57 us; skew 500 ppm
I20260925 05:11:27.277602 25853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:27 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313087, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:27.280846 25853 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:11:27.282399 25853 webserver.cc:460] Webserver started at http://127.8.215.2:39995/ using document root <none> and password file <none>
I20260925 05:11:27.283170 25853 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:27.283319 25853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:27.289212 25853 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:11:27.292026 25868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:27.293457 25853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:27.293639 25853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "ed39eeb61eb74af3a53a9ab6e27b5fb2"
format_stamp: "Formatted at 2026-09-25 05:10:41 on dist-test-slave-wpdm"
server_key: "0e59a88f933536b08a9bb62ea7258632"
server_key_iv: "333ebbb9c3dfe7cbc661f165bad02edf"
server_key_version: "encryptionkey@0"
I20260925 05:11:27.294663 25853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:27.307685 25853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:27.311748 25853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:27.312042 25853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:27.313083 25853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:27.314464 25853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:27.314584 25853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:27.314728 25853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:27.314819 25853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:27.348666 25853 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:44793
I20260925 05:11:27.348790 25980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:44793 every 8 connection(s)
I20260925 05:11:27.350216 25853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:27.355026 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.353172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:36153)
0925 05:11:27.353881 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.354108 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
W20260925 05:11:27.355692 25981 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:44205 (0 consecutive failures): Network error: Failed to ping master at 127.8.215.62:44205: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
I20260925 05:11:27.356491 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.356184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:44869)
0925 05:11:27.356315 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.356384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:11:27.357348 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.357059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:55381)
0925 05:11:27.357174 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.357248 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 05:11:27.357602 25981 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:11:27.359753 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25853
I20260925 05:11:27.360353 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:45163
--local_ip_for_outbound_sockets=127.8.215.3
--tserver_master_addrs=127.8.215.62:44205
--webserver_port=37201
--webserver_interface=127.8.215.3
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:27.686486 25984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:27.686858 25984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:27.687022 25984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:27.695008 25984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:27.695230 25984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:27.695429 25984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:11:27.706120 25984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:45163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=37201
--tserver_master_addrs=127.8.215.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:27.707540 25984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:27.708962 25984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:27.723097 25992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:27.723076 25990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:27.723192 25989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:27.724036 25984 server_base.cc:1034] running on GCE node
I20260925 05:11:27.724900 25984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:27.725831 25984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:27.727074 25984 hybrid_clock.cc:648] HybridClock initialized: now 1790313087727028 us; error 59 us; skew 500 ppm
I20260925 05:11:27.727447 25984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:27 dist-test-slave-wpdm krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313087, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:27.730702 25984 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:11:27.732185 25984 webserver.cc:460] Webserver started at http://127.8.215.3:37201/ using document root <none> and password file <none>
I20260925 05:11:27.733014 25984 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:27.733165 25984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:27.738767 25984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:11:27.741644 25999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:27.743080 25984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:27.743291 25984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "7571025ed4cd4a13abe5ca860a781e8d"
format_stamp: "Formatted at 2026-09-25 05:10:42 on dist-test-slave-wpdm"
server_key: "368818a203fbd8ef2910007a2ed90aa7"
server_key_iv: "645c4694093dd0c04bbf2cec0b4ff9de"
server_key_version: "encryptionkey@0"
I20260925 05:11:27.744324 25984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:27.761327 25984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:27.765730 25984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:27.766011 25984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:27.766978 25984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:27.768308 25984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:27.768426 25984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:27.768559 25984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:27.768651 25984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:27.803328 25984 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:45163
I20260925 05:11:27.803375 26111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:45163 every 8 connection(s)
I20260925 05:11:27.804862 25984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:27.805455 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 25984
I20260925 05:11:27.810103 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.808091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:48725)
0925 05:11:27.808980 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.809185 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":351,"threads_started":1}
W20260925 05:11:27.810765 26112 heartbeater.cc:644] Failed to heartbeat to 127.8.215.62:44205 (0 consecutive failures): Network error: Failed to ping master at 127.8.215.62:44205: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
I20260925 05:11:27.811592 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.811236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:47935)
0925 05:11:27.811435 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.811497 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 05:11:27.812364 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.812047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:44843)
0925 05:11:27.812183 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.812267 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:11:27.812609 26112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:11:27.882046 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:27.881079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:38855)
0925 05:11:27.881641 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:27.881791 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":283,"threads_started":1}
I20260925 05:11:28.358872 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:28.358094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:49959)
0925 05:11:28.358567 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:28.358648 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260925 05:11:28.814096 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:28.813264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:51667)
0925 05:11:28.813726 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:28.813836 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260925 05:11:28.884147 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:28.883214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:57547)
0925 05:11:28.883876 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:28.883974 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":294,"threads_started":1}
I20260925 05:11:29.360452 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:29.359735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:39143)
0925 05:11:29.360189 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:29.360279 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":224,"threads_started":1}
I20260925 05:11:29.815806 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:29.815107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:57761)
0925 05:11:29.815554 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:29.815640 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":218,"threads_started":1}
I20260925 05:11:29.886116 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:29.885213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:36883)
0925 05:11:29.885837 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:29.885940 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":324,"threads_started":1}
I20260925 05:11:30.362144 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:30.361433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:38327)
0925 05:11:30.361870 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:30.361975 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260925 05:11:30.817521 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:30.816705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:36249)
0925 05:11:30.817267 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:30.817354 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":311,"threads_started":1}
I20260925 05:11:30.887943 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:30.887095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:33443)
0925 05:11:30.887595 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:30.887695 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
I20260925 05:11:31.363835 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:31.363065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:51851)
0925 05:11:31.363547 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:31.363626 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260925 05:11:31.819207 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:31.818466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:43969)
0925 05:11:31.818931 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:31.819018 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260925 05:11:31.889714 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:31.888991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:53669)
0925 05:11:31.889455 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:31.889542 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260925 05:11:32.365522 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:32.364832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:37511)
0925 05:11:32.365260 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:32.365340 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260925 05:11:32.820990 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:32.820098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:47239)
0925 05:11:32.820613 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:32.820706 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":264,"threads_started":1}
I20260925 05:11:32.891436 26131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:32.890715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:37383)
0925 05:11:32.891178 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:32.891267 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260925 05:11:33.367158 26132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:33.366422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:54267)
0925 05:11:33.366874 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:33.366987 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260925 05:11:33.822548 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:33.821829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:33253)
0925 05:11:33.822286 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:33.822373 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260925 05:11:33.893326 26134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:33.892366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:47503)
0925 05:11:33.893000 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:33.893106 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":237,"threads_started":1}
I20260925 05:11:34.369102 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:34.368224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:59305)
0925 05:11:34.368718 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:34.368877 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":220,"threads_started":1}
I20260925 05:11:34.824280 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:34.823529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:45711)
0925 05:11:34.823995 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:34.824072 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":203,"threads_started":1}
I20260925 05:11:34.894995 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:34.894286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:45225)
0925 05:11:34.894704 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:34.894810 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260925 05:11:35.370903 26138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:35.370020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:35943)
0925 05:11:35.370543 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:35.370676 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":267,"threads_started":1}
I20260925 05:11:35.825932 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:35.825241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:52447)
0925 05:11:35.825688 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:35.825772 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260925 05:11:35.896903 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:35.895962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:42469)
0925 05:11:35.896522 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:35.896646 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":329,"threads_started":1}
I20260925 05:11:36.372697 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:36.371946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:55675)
0925 05:11:36.372439 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:36.372521 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":231,"threads_started":1}
I20260925 05:11:36.827601 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:36.826875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:48229)
0925 05:11:36.827347 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:36.827431 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":216,"threads_started":1}
I20260925 05:11:36.898882 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:36.897918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:44191)
0925 05:11:36.898489 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:36.898572 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":338,"threads_started":1}
I20260925 05:11:37.374608 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:37.373830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:56013)
0925 05:11:37.374355 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:37.374443 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260925 05:11:37.829401 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:37.828649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:39727)
0925 05:11:37.829130 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:37.829229 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":290,"threads_started":1}
I20260925 05:11:37.900717 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:37.899918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:43117)
0925 05:11:37.900411 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:37.900507 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
I20260925 05:11:38.376508 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:38.375652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:57033)
0925 05:11:38.376214 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:38.376326 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":300,"threads_started":1}
I20260925 05:11:38.831316 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:38.830359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:38953)
0925 05:11:38.830941 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:38.831064 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260925 05:11:38.902473 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:38.901817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:59609)
0925 05:11:38.902247 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:38.902321 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260925 05:11:39.378335 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:39.377504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:47129)
0925 05:11:39.378010 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:39.378129 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":223,"threads_started":1}
I20260925 05:11:39.833379 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:39.832455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:40001)
0925 05:11:39.833043 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:39.833189 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":303,"threads_started":1}
I20260925 05:11:39.904081 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:39.903384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:42567)
0925 05:11:39.903807 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:39.903900 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260925 05:11:40.380097 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:40.379320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:40359)
0925 05:11:40.379789 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:40.379923 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":219,"threads_started":1}
I20260925 05:11:40.834961 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:40.834198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:39159)
0925 05:11:40.834665 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:40.834769 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":234,"threads_started":1}
I20260925 05:11:40.905658 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:40.904983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:55995)
0925 05:11:40.905400 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:40.905478 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":203,"threads_started":1}
I20260925 05:11:41.381819 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:41.381046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:52133)
0925 05:11:41.381555 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:41.381639 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":249,"threads_started":1}
I20260925 05:11:41.836586 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:41.835825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:48339)
0925 05:11:41.836319 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:41.836413 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":237,"threads_started":1}
I20260925 05:11:41.907508 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:41.906783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.1:33661)
0925 05:11:41.907224 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:41.907325 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260925 05:11:42.383666 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:42.382949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.2:55397)
0925 05:11:42.383382 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:42.383490 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":213,"threads_started":1}
I20260925 05:11:42.807055 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25682
I20260925 05:11:42.829970 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:42.808148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.8.215.1:37367)
0925 05:11:42.808768 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:42.808789 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:42.809162 (+ 373us) server_negotiation.cc:407] Connection header received
0925 05:11:42.810810 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:42.810818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:42.811252 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:42.811540 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:42.812940 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:42.814174 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:42.815426 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:42.816108 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:42.816239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:42.824139 (+ 7900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:42.824147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:42.824164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:42.824215 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:42.826947 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:42.827606 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:42.827613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:42.827620 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:42.827775 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:42.828080 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:42.828083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:42.828085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:42.828581 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:42.828882 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:42.829262 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 05:11:42.829670 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":268,"threads_started":1}
I20260925 05:11:42.831349 25845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:42.838601 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:42.837753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44205 (local address 127.8.215.3:60159)
0925 05:11:42.838243 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:42.838360 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.62:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260925 05:11:42.916882 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25682
I20260925 05:11:42.935956 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25853
I20260925 05:11:42.956638 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:42.937706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.8.215.2:44793)
0925 05:11:42.938461 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:42.938495 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:42.938542 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:11:42.940925 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:42.940943 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:42.941427 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:42.941786 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:42.943560 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:42.945454 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:42.946738 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:42.947411 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:42.947597 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:42.950493 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:42.950501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:42.950517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:42.950553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:42.953448 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:42.954109 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:42.954119 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:42.954131 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:42.954293 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:42.954661 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:42.954667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:42.954670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:42.955176 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:42.955474 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:42.955942 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 05:11:42.956350 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":361,"threads_started":1}
I20260925 05:11:42.958001 25976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:43.041205 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25853
I20260925 05:11:43.059475 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 25984
I20260925 05:11:43.076993 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:43.060579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.8.215.3:45163)
0925 05:11:43.061316 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:43.061329 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:43.061370 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:11:43.062933 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:43.062943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:43.063355 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:43.063633 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:43.065047 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:43.066304 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:43.067608 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:43.068275 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:43.068454 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:43.071013 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:43.071021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:43.071036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:43.071069 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:43.073914 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:43.074562 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:43.074568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:43.074574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:43.074730 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:43.075130 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:43.075133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:43.075135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:43.075638 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:43.075873 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:43.076283 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 05:11:43.076619 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":416,"threads_started":1}
I20260925 05:11:43.078380 26107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:43.162757 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 25984
2026-09-25T05:11:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64408 ms)
[----------] 30 tests from SecurityITest (301199 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:11:43 dist-test-slave-wpdm krb5kdc[26170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:11:43 dist-test-slave-wpdm krb5kdc[26170](info): set up 2 sockets
Sep 25 05:11:43 dist-test-slave-wpdm krb5kdc[26170](info): commencing operation
krb5kdc: starting...
W20260925 05:11:44.289417 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:11:44 dist-test-slave-wpdm krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:11:44.548281 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:41871
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:44.880905 26183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:44.881246 26183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:44.881403 26183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:44.889180 26183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:44.889338 26183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:44.889462 26183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:44.889568 26183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:44.889655 26183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:44.900743 26183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:41871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:44.902125 26183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:44.903415 26183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:44.915426 26191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:44.915426 26189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:44.915477 26188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:44.916159 26183 server_base.cc:1034] running on GCE node
I20260925 05:11:44.917011 26183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:44.918388 26183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:44.919607 26183 hybrid_clock.cc:648] HybridClock initialized: now 1790313104919569 us; error 50 us; skew 500 ppm
I20260925 05:11:44.919936 26183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:44 dist-test-slave-wpdm krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:44.922945 26183 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:11:44.924307 26183 webserver.cc:460] Webserver started at http://127.8.215.62:41237/ using document root <none> and password file <none>
I20260925 05:11:44.925037 26183 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:44.925173 26183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:44.925479 26183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:44.928385 26183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "7ee994c8cb8e4321973d8b0c3eec42f8"
format_stamp: "Formatted at 2026-09-25 05:11:44 on dist-test-slave-wpdm"
server_key: "01c7dc592333d9ca5efcde1f6074dba2"
server_key_iv: "24b04b340f128063a96b721e7d2ffe0c"
server_key_version: "encryptionkey@0"
I20260925 05:11:44.929137 26183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "7ee994c8cb8e4321973d8b0c3eec42f8"
format_stamp: "Formatted at 2026-09-25 05:11:44 on dist-test-slave-wpdm"
server_key: "01c7dc592333d9ca5efcde1f6074dba2"
server_key_iv: "24b04b340f128063a96b721e7d2ffe0c"
server_key_version: "encryptionkey@0"
I20260925 05:11:44.934037 26183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:11:44.939031 26198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:44.940991 26183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 05:11:44.941229 26183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "7ee994c8cb8e4321973d8b0c3eec42f8"
format_stamp: "Formatted at 2026-09-25 05:11:44 on dist-test-slave-wpdm"
server_key: "01c7dc592333d9ca5efcde1f6074dba2"
server_key_iv: "24b04b340f128063a96b721e7d2ffe0c"
server_key_version: "encryptionkey@0"
I20260925 05:11:44.941427 26183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:44.973165 26183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:44.977910 26183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:44.978163 26183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:44.995477 26183 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:41871
I20260925 05:11:44.995510 26249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:41871 every 8 connection(s)
I20260925 05:11:44.996870 26183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:45.000610 26250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:45.003454 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26183
I20260925 05:11:45.003674 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:11:45.004370 9052 external_mini_cluster.cc:1428] Setting key 2bedf6730919f3e074d6f4354a5ef188
I20260925 05:11:45.012985 26250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8: Bootstrap starting.
I20260925 05:11:45.016788 26250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:45.018136 26250 log.cc:824] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:11:45 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:45.021111 26250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8: No bootstrap required, opened a new log
I20260925 05:11:45.028035 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:45.006352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.8.215.62:41871)
0925 05:11:45.007138 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:45.007149 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:45.007189 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:11:45.008877 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:45.008899 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:45.009296 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:45.009608 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:45.011434 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:45.012682 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:45.014594 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:45.015278 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:45.015429 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:45.020570 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:45.020577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:45.020591 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:45.020624 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:45.023849 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:45.024643 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:45.024650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:45.024660 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:45.024887 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:45.025265 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:45.025270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:45.025274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:45.025812 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:45.026147 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:45.026570 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 05:11:45.027026 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":373,"threads_started":1}
I20260925 05:11:45.029245 26250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41871 } }
I20260925 05:11:45.029620 26250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:45.029749 26250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee994c8cb8e4321973d8b0c3eec42f8, State: Initialized, Role: FOLLOWER
I20260925 05:11:45.030370 26250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41871 } }
I20260925 05:11:45.030618 26250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:45.030759 26250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:45.030947 26250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:45.033182 26250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41871 } }
I20260925 05:11:45.033671 26250 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ee994c8cb8e4321973d8b0c3eec42f8; no voters:
I20260925 05:11:45.034261 26250 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:45.034425 26255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:45.036208 26255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ee994c8cb8e4321973d8b0c3eec42f8, State: Running, Role: LEADER
I20260925 05:11:45.036630 26250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:45.036952 26255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41871 } }
I20260925 05:11:45.039768 26256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41871 } } }
I20260925 05:11:45.040185 26256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:45.040612 26257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee994c8cb8e4321973d8b0c3eec42f8. Latest consensus state: current_term: 1 leader_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee994c8cb8e4321973d8b0c3eec42f8" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 41871 } } }
I20260925 05:11:45.040928 26257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:45.044628 26262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:45.049286 26262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:45.057883 26262 catalog_manager.cc:1347] Generated new cluster ID: 2174e036b144493d9255e0b7311f4f69
I20260925 05:11:45.057992 26262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:45.068871 26262 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:45.070160 26262 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:45.082283 26262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8: Generated new TSK 0
I20260925 05:11:45.083379 26262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:11:45.151940 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41871
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:45.486982 26276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:45.487257 26276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:45.487414 26276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:45.494472 26276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:45.494572 26276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:45.494771 26276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:11:45.504241 26276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:45.505513 26276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:45.506785 26276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:45.519660 26281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:45.519618 26282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:45.519830 26284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:45.522061 26276 server_base.cc:1034] running on GCE node
I20260925 05:11:45.522681 26276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:45.523645 26276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:45.524822 26276 hybrid_clock.cc:648] HybridClock initialized: now 1790313105524767 us; error 45 us; skew 500 ppm
I20260925 05:11:45.525147 26276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:45 dist-test-slave-wpdm krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313105, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:45.528249 26276 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:11:45.529700 26276 webserver.cc:460] Webserver started at http://127.8.215.1:43243/ using document root <none> and password file <none>
I20260925 05:11:45.530356 26276 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:45.530452 26276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:45.530689 26276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:45.533528 26276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "488bb8a16c1147c0a8c275413fd48496"
format_stamp: "Formatted at 2026-09-25 05:11:45 on dist-test-slave-wpdm"
server_key: "aaac5c6d54b3170a5866ee548ec87513"
server_key_iv: "da5ab381d9c1105373d20b2880f18e16"
server_key_version: "encryptionkey@0"
I20260925 05:11:45.534241 26276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "488bb8a16c1147c0a8c275413fd48496"
format_stamp: "Formatted at 2026-09-25 05:11:45 on dist-test-slave-wpdm"
server_key: "aaac5c6d54b3170a5866ee548ec87513"
server_key_iv: "da5ab381d9c1105373d20b2880f18e16"
server_key_version: "encryptionkey@0"
I20260925 05:11:45.539494 26276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:11:45.542898 26291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:45.544257 26276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:11:45.544391 26276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "488bb8a16c1147c0a8c275413fd48496"
format_stamp: "Formatted at 2026-09-25 05:11:45 on dist-test-slave-wpdm"
server_key: "aaac5c6d54b3170a5866ee548ec87513"
server_key_iv: "da5ab381d9c1105373d20b2880f18e16"
server_key_version: "encryptionkey@0"
I20260925 05:11:45.544544 26276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:45.559177 26276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:45.563117 26276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:45.563398 26276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:45.564262 26276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:45.565572 26276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:45.565641 26276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:45.565729 26276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:45.565775 26276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:45.598713 26276 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:36601
I20260925 05:11:45.598819 26403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:36601 every 8 connection(s)
I20260925 05:11:45.600016 26276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:45.609280 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26276
I20260925 05:11:45.609488 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:45.610100 9052 external_mini_cluster.cc:1428] Setting key 808676477e993d20724cc47ea4e25f39
Sep 25 05:11:45 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313105, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:45.619783 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:45.602525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:48507 (local address 127.8.215.62:41871)
0925 05:11:45.602962 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:45.602967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:45.603963 (+ 996us) server_negotiation.cc:407] Connection header received
0925 05:11:45.605915 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:45.605920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:45.606016 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:45.606138 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:45.607638 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:45.608392 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:45.610107 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:45.610575 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:45.610708 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:45.614083 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:45.614088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:45.614091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:45.614121 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:45.616705 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:45.617491 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:45.617497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:45.617501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:45.617624 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:45.618048 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:45.618053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:45.618057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:45.618330 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:45.618494 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:45.619192 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 05:11:45.619416 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":187,"threads_started":1}
I20260925 05:11:45.620801 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:45.603082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:41871 (local address 127.8.215.1:48507)
0925 05:11:45.603806 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:45.603847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:45.605643 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:45.606383 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:45.606391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:45.606914 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:45.607466 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:45.607482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:45.608555 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:45.608559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:45.609964 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:45.609976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:45.610796 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:45.610805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:45.611065 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:45.611732 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:45.611754 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:45.613848 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:45.616898 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:45.616907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:45.616924 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:45.617367 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:45.617728 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:45.617733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:45.617737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:45.617934 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:45.618454 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:45.618462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:45.618809 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 05:11:45.619068 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 05:11:45.619597 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":302,"threads_started":1}
I20260925 05:11:45.622416 26404 heartbeater.cc:347] Connected to a master server at 127.8.215.62:41871
I20260925 05:11:45.622730 26404 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:45.623623 26404 heartbeater.cc:511] Master 127.8.215.62:41871 requested a full tablet report, sending...
I20260925 05:11:45.626051 26215 ts_manager.cc:194] Registered new tserver with Master: 488bb8a16c1147c0a8c275413fd48496 (127.8.215.1:36601)
I20260925 05:11:45.628283 26215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:48507
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:11:45.669366 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41871
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:45.999775 26410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:46.000053 26410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:46.000231 26410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:46.007954 26410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:46.008075 26410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:46.008279 26410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:11:46.018327 26410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:46.019625 26410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:46.021018 26410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:46.034528 26416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:46.034531 26418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:46.034869 26415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:46.035384 26410 server_base.cc:1034] running on GCE node
I20260925 05:11:46.035988 26410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:46.036998 26410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:46.038206 26410 hybrid_clock.cc:648] HybridClock initialized: now 1790313106038165 us; error 39 us; skew 500 ppm
I20260925 05:11:46.038525 26410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:46 dist-test-slave-wpdm krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313106, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:46.041615 26410 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:11:46.043149 26410 webserver.cc:460] Webserver started at http://127.8.215.2:40967/ using document root <none> and password file <none>
I20260925 05:11:46.043809 26410 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:46.043942 26410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:46.044250 26410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:46.047354 26410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "3e92008215e3445791b89b2f9d5a15e6"
format_stamp: "Formatted at 2026-09-25 05:11:46 on dist-test-slave-wpdm"
server_key: "f5977ad77dabac2d2850fb9ec3425e5b"
server_key_iv: "de905bb6c6e753a4c9bb42f74ccab1bd"
server_key_version: "encryptionkey@0"
I20260925 05:11:46.048233 26410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "3e92008215e3445791b89b2f9d5a15e6"
format_stamp: "Formatted at 2026-09-25 05:11:46 on dist-test-slave-wpdm"
server_key: "f5977ad77dabac2d2850fb9ec3425e5b"
server_key_iv: "de905bb6c6e753a4c9bb42f74ccab1bd"
server_key_version: "encryptionkey@0"
I20260925 05:11:46.053314 26410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:11:46.056576 26425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:46.058058 26410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:46.058219 26410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "3e92008215e3445791b89b2f9d5a15e6"
format_stamp: "Formatted at 2026-09-25 05:11:46 on dist-test-slave-wpdm"
server_key: "f5977ad77dabac2d2850fb9ec3425e5b"
server_key_iv: "de905bb6c6e753a4c9bb42f74ccab1bd"
server_key_version: "encryptionkey@0"
I20260925 05:11:46.058418 26410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:46.068859 26410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:46.072755 26410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:46.073060 26410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:46.073994 26410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:46.075258 26410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:46.075356 26410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:46.075498 26410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:46.075565 26410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:46.108384 26410 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:37289
I20260925 05:11:46.108407 26537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:37289 every 8 connection(s)
I20260925 05:11:46.109741 26410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:46.113756 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26410
I20260925 05:11:46.113977 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:46.114424 9052 external_mini_cluster.cc:1428] Setting key dfbd50fd57818607027ad1b4e9687471
Sep 25 05:11:46 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313106, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:46.129350 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.112171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:42703 (local address 127.8.215.62:41871)
0925 05:11:46.112362 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.112367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.113528 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 05:11:46.115602 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.115608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.115699 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.115804 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:46.117374 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.118060 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.119597 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.120040 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.120164 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.123696 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:46.123701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:46.123704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:46.123733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:46.126447 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.127123 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.127132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.127135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.127255 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.127669 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.127674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.127677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.127940 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:46.128104 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.128775 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.129010 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:11:46.130255 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.112643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:41871 (local address 127.8.215.2:42703)
0925 05:11:46.113376 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.113413 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.115335 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.116052 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.116062 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.116583 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:46.117204 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.117222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.118207 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.118212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.119461 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.119472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.120263 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.120271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.120495 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.121190 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:46.121212 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:46.123482 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:46.126579 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:46.126587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:46.126603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:46.127002 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:46.127367 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:46.127372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:46.127375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:46.127552 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:46.128063 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:46.128071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:46.128392 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.128640 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.129166 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":261,"threads_started":1}
I20260925 05:11:46.131672 26538 heartbeater.cc:347] Connected to a master server at 127.8.215.62:41871
I20260925 05:11:46.132028 26538 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:46.132882 26538 heartbeater.cc:511] Master 127.8.215.62:41871 requested a full tablet report, sending...
I20260925 05:11:46.134447 26215 ts_manager.cc:194] Registered new tserver with Master: 3e92008215e3445791b89b2f9d5a15e6 (127.8.215.2:37289)
I20260925 05:11:46.135241 26215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:42703
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:11:46.172236 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41871
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:46.519862 26543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:46.520164 26543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:46.520267 26543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:46.527875 26543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:46.527992 26543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:46.528131 26543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:11:46.538383 26543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:41871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:46.539662 26543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:46.540982 26543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:46.554507 26548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:46.554447 26551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:46.554481 26549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:46.555279 26543 server_base.cc:1034] running on GCE node
I20260925 05:11:46.555872 26543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:46.556911 26543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:46.558120 26543 hybrid_clock.cc:648] HybridClock initialized: now 1790313106558076 us; error 42 us; skew 500 ppm
I20260925 05:11:46.558427 26543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:46 dist-test-slave-wpdm krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313106, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:46.561688 26543 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:11:46.563237 26543 webserver.cc:460] Webserver started at http://127.8.215.3:45747/ using document root <none> and password file <none>
I20260925 05:11:46.563901 26543 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:46.564045 26543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:46.564356 26543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:46.567319 26543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "bc843959d0694e22a87ec369349ab148"
format_stamp: "Formatted at 2026-09-25 05:11:46 on dist-test-slave-wpdm"
server_key: "92aca4f27353b3d0afc41b838b995110"
server_key_iv: "8ae5f8c95e20a198c74154d28e0a533d"
server_key_version: "encryptionkey@0"
I20260925 05:11:46.568063 26543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "bc843959d0694e22a87ec369349ab148"
format_stamp: "Formatted at 2026-09-25 05:11:46 on dist-test-slave-wpdm"
server_key: "92aca4f27353b3d0afc41b838b995110"
server_key_iv: "8ae5f8c95e20a198c74154d28e0a533d"
server_key_version: "encryptionkey@0"
I20260925 05:11:46.572978 26543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:11:46.576233 26558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:46.577504 26543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:46.577665 26543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "bc843959d0694e22a87ec369349ab148"
format_stamp: "Formatted at 2026-09-25 05:11:46 on dist-test-slave-wpdm"
server_key: "92aca4f27353b3d0afc41b838b995110"
server_key_iv: "8ae5f8c95e20a198c74154d28e0a533d"
server_key_version: "encryptionkey@0"
I20260925 05:11:46.577859 26543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:46.604799 26543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:46.609275 26543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:46.609571 26543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:46.610517 26543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:46.611807 26543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:46.611904 26543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:46.612038 26543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:46.612104 26543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:46.632136 26404 heartbeater.cc:503] Master 127.8.215.62:41871 was elected leader, sending a full tablet report...
I20260925 05:11:46.644933 26543 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:35661
I20260925 05:11:46.645025 26670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:35661 every 8 connection(s)
I20260925 05:11:46.646271 26543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:46.648120 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26543
I20260925 05:11:46.648320 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:46.648698 9052 external_mini_cluster.cc:1428] Setting key b8868ed8597999fa85ee31a9a1b37b3a
Sep 25 05:11:46 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313106, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:46.666949 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.649999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:37869 (local address 127.8.215.62:41871)
0925 05:11:46.653071 (+ 3072us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.653075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.653091 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:11:46.653468 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.653472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.653538 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.653622 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:46.655007 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.655757 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.657260 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.657636 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.657729 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.660945 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:46.660955 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:46.660958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:46.660987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:46.663902 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.664693 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.664698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.664702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.664877 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.665287 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.665292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.665295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.665560 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:46.665727 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.666372 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.666579 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2973,"thread_start_us":2883,"threads_started":1}
I20260925 05:11:46.667851 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.650594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:41871 (local address 127.8.215.3:37869)
0925 05:11:46.651475 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.651522 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.653242 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.653810 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.653818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.654342 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:46.654874 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.654890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.655871 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.655874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.657143 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.657155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.657806 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.657810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.658006 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.658588 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:46.658609 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:46.660695 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:46.664024 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:46.664033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:46.664049 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:46.664573 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:46.664981 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:46.664986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:46.664989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:46.665169 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:46.665690 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:46.665698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:46.666018 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.666270 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.666781 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":356,"threads_started":1}
I20260925 05:11:46.669116 26671 heartbeater.cc:347] Connected to a master server at 127.8.215.62:41871
I20260925 05:11:46.669404 26671 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:46.670040 26671 heartbeater.cc:511] Master 127.8.215.62:41871 requested a full tablet report, sending...
I20260925 05:11:46.671314 26215 ts_manager.cc:194] Registered new tserver with Master: bc843959d0694e22a87ec369349ab148 (127.8.215.3:35661)
I20260925 05:11:46.672104 26215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:37869
I20260925 05:11:46.676543 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:46.693429 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.681077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.8.215.62:41871)
0925 05:11:46.681280 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.681288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.681712 (+ 424us) server_negotiation.cc:407] Connection header received
0925 05:11:46.681931 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.681936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.682007 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.682130 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:46.683569 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.684291 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.685612 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.686016 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.686149 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.688415 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:46.688426 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:46.688430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:46.688459 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:46.690803 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.691463 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.691470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.691473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.691593 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.691967 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.691973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.691976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.692289 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:46.692537 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.692904 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.693116 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:11:46.697062 26215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:46.703696 26215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:46.737426 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.716497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.8.215.2:37289)
0925 05:11:46.728936 (+ 12439us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.728947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.728978 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:11:46.729075 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.729084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.729331 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.730354 (+ 1023us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:46.730575 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.732221 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.734934 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.736090 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.736235 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.736490 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.736790 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.737020 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12255,"thread_start_us":13068,"threads_started":1}
I20260925 05:11:46.737584 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.715425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:36601 (local address 127.0.0.1:49916)
0925 05:11:46.716181 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.716222 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.716414 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.729670 (+ 13256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.729677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.729742 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:46.730371 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.730384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.732365 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.732371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.734837 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.734870 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.736438 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.736444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.736579 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.736603 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.736693 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.737019 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":238,"spinlock_wait_cycles":7424,"thread_start_us":344,"threads_started":1}
I20260925 05:11:46.738166 26686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.715949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:37289 (local address 127.0.0.1:57856)
0925 05:11:46.717088 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.717110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.717247 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.729589 (+ 12342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.729598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.729628 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:46.730094 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.730107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.732406 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.732409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.734774 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.734789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.736292 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.736300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.736529 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.736584 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.736743 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.737019 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":72,"spinlock_wait_cycles":1792,"thread_start_us":862,"threads_started":1}
I20260925 05:11:46.738416 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.716698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:35661 (local address 127.0.0.1:49124)
0925 05:11:46.727668 (+ 10970us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.727695 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.727870 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.731284 (+ 3414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.731289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.731314 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:46.731659 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.731668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.733441 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.733446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.735570 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.735585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.737492 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.737499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.737700 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.737749 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.737840 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.737981 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10796,"thread_start_us":12657,"threads_started":1}
I20260925 05:11:46.738722 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.717086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.8.215.3:35661)
0925 05:11:46.730846 (+ 13760us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.730851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.730876 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:11:46.730948 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.730952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.731090 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.731299 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:46.731848 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.733290 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.735734 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.737101 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.737227 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.737465 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.738169 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.738360 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13605,"thread_start_us":13453,"threads_started":1}
I20260925 05:11:46.739171 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.715624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.8.215.1:36601)
0925 05:11:46.728598 (+ 12974us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.728611 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.728651 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:11:46.728802 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.728813 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.729181 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.730125 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:46.730550 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.732222 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.734993 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.736217 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.738294 (+ 2077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.738573 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.738652 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.738823 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12734,"thread_start_us":12539,"threads_started":1}
I20260925 05:11:46.742714 26339 tablet_service.cc:1467] Processing CreateTablet for tablet f056134e3ec34c98a98f7e7d3c668198 (DEFAULT_TABLE table=test-table [id=cb33554f6b1849a1b3e36bf31a0127da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:46.742798 26606 tablet_service.cc:1467] Processing CreateTablet for tablet f056134e3ec34c98a98f7e7d3c668198 (DEFAULT_TABLE table=test-table [id=cb33554f6b1849a1b3e36bf31a0127da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:46.742798 26473 tablet_service.cc:1467] Processing CreateTablet for tablet f056134e3ec34c98a98f7e7d3c668198 (DEFAULT_TABLE table=test-table [id=cb33554f6b1849a1b3e36bf31a0127da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:46.744065 26339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f056134e3ec34c98a98f7e7d3c668198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:46.744123 26606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f056134e3ec34c98a98f7e7d3c668198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:46.744123 26473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f056134e3ec34c98a98f7e7d3c668198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:46.755126 26690 tablet_bootstrap.cc:492] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Bootstrap starting.
I20260925 05:11:46.758600 26691 tablet_bootstrap.cc:492] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Bootstrap starting.
I20260925 05:11:46.759384 26690 tablet_bootstrap.cc:654] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:46.760766 26690 log.cc:824] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:46.762151 26691 tablet_bootstrap.cc:654] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:46.763417 26691 log.cc:824] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:46.764078 26690 tablet_bootstrap.cc:492] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: No bootstrap required, opened a new log
I20260925 05:11:46.764151 26692 tablet_bootstrap.cc:492] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Bootstrap starting.
I20260925 05:11:46.764353 26690 ts_tablet_manager.cc:1397] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 05:11:46.766031 26691 tablet_bootstrap.cc:492] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: No bootstrap required, opened a new log
I20260925 05:11:46.766309 26691 ts_tablet_manager.cc:1397] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260925 05:11:46.767539 26692 tablet_bootstrap.cc:654] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:46.768812 26692 log.cc:824] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:46.771909 26692 tablet_bootstrap.cc:492] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: No bootstrap required, opened a new log
I20260925 05:11:46.772166 26692 ts_tablet_manager.cc:1397] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 05:11:46.774078 26691 raft_consensus.cc:348] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.774457 26691 raft_consensus.cc:374] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:46.774557 26691 raft_consensus.cc:729] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e92008215e3445791b89b2f9d5a15e6, State: Initialized, Role: FOLLOWER
I20260925 05:11:46.774633 26690 raft_consensus.cc:348] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.775056 26690 raft_consensus.cc:374] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:46.775154 26690 raft_consensus.cc:729] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488bb8a16c1147c0a8c275413fd48496, State: Initialized, Role: FOLLOWER
I20260925 05:11:46.775220 26691 consensus_queue.cc:260] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.775730 26690 consensus_queue.cc:260] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.776473 26538 heartbeater.cc:503] Master 127.8.215.62:41871 was elected leader, sending a full tablet report...
I20260925 05:11:46.777029 26691 ts_tablet_manager.cc:1428] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 05:11:46.784305 26692 raft_consensus.cc:348] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.784812 26692 raft_consensus.cc:374] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:46.784910 26692 raft_consensus.cc:729] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc843959d0694e22a87ec369349ab148, State: Initialized, Role: FOLLOWER
I20260925 05:11:46.785506 26692 consensus_queue.cc:260] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.786547 26671 heartbeater.cc:503] Master 127.8.215.62:41871 was elected leader, sending a full tablet report...
I20260925 05:11:46.787133 26692 ts_tablet_manager.cc:1428] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260925 05:11:46.787561 26690 ts_tablet_manager.cc:1428] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Time spent starting tablet: real 0.023s user 0.013s sys 0.006s
I20260925 05:11:46.851473 26696 raft_consensus.cc:482] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:46.851794 26696 raft_consensus.cc:504] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
W20260925 05:11:46.852684 26405 tablet.cc:2369] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:46.853807 26696 leader_election.cc:288] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 488bb8a16c1147c0a8c275413fd48496 (127.8.215.1:36601), bc843959d0694e22a87ec369349ab148 (127.8.215.3:35661)
I20260925 05:11:46.861953 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.854165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:36601 (local address 127.8.215.2:46669)
0925 05:11:46.854817 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.854836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.855003 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.855361 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.855366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.855401 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:46.855806 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.855819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.857089 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.857094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.859230 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.859246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.861152 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.861159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.861340 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.861380 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.861483 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.861618 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":349,"threads_started":1}
I20260925 05:11:46.862447 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.854146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:46669 (local address 127.8.215.1:36601)
0925 05:11:46.854369 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.854379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.854927 (+ 548us) server_negotiation.cc:407] Connection header received
0925 05:11:46.855136 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.855141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.855248 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.855339 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:46.855951 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.856973 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.859431 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.860775 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.860904 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.861008 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.861557 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.862199 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:11:46.862469 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.855072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:43073 (local address 127.8.215.3:35661)
0925 05:11:46.855273 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.855280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.855689 (+ 409us) server_negotiation.cc:407] Connection header received
0925 05:11:46.855828 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.855833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.855913 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.855991 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:46.856566 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.857501 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.860192 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.861436 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.861551 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.861641 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.862161 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.862299 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 05:11:46.861956 26539 tablet.cc:2369] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:46.862825 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.854935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:35661 (local address 127.8.215.2:43073)
0925 05:11:46.855595 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:46.855608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:46.855711 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:46.856035 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:46.856041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:46.856068 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:46.856418 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.856431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.857660 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.857683 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.859972 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:46.859986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.861663 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.861669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:46.861878 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.861924 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 05:11:46.862123 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 05:11:46.862391 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":3,"thread_start_us":376,"threads_started":1}
I20260925 05:11:46.863873 26626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f056134e3ec34c98a98f7e7d3c668198" candidate_uuid: "3e92008215e3445791b89b2f9d5a15e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc843959d0694e22a87ec369349ab148" is_pre_election: true
I20260925 05:11:46.863897 26359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f056134e3ec34c98a98f7e7d3c668198" candidate_uuid: "3e92008215e3445791b89b2f9d5a15e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488bb8a16c1147c0a8c275413fd48496" is_pre_election: true
I20260925 05:11:46.864444 26626 raft_consensus.cc:2454] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e92008215e3445791b89b2f9d5a15e6 in term 0.
I20260925 05:11:46.864485 26359 raft_consensus.cc:2454] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e92008215e3445791b89b2f9d5a15e6 in term 0.
I20260925 05:11:46.865104 26426 leader_election.cc:302] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e92008215e3445791b89b2f9d5a15e6, 488bb8a16c1147c0a8c275413fd48496; no voters:
I20260925 05:11:46.865475 26696 raft_consensus.cc:2781] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:46.865628 26696 raft_consensus.cc:482] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:46.865729 26696 raft_consensus.cc:3037] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:46.867857 26696 raft_consensus.cc:504] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.868409 26696 leader_election.cc:288] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [CANDIDATE]: Term 1 election: Requested vote from peers 488bb8a16c1147c0a8c275413fd48496 (127.8.215.1:36601), bc843959d0694e22a87ec369349ab148 (127.8.215.3:35661)
I20260925 05:11:46.868854 26359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f056134e3ec34c98a98f7e7d3c668198" candidate_uuid: "3e92008215e3445791b89b2f9d5a15e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488bb8a16c1147c0a8c275413fd48496"
I20260925 05:11:46.868920 26626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f056134e3ec34c98a98f7e7d3c668198" candidate_uuid: "3e92008215e3445791b89b2f9d5a15e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc843959d0694e22a87ec369349ab148"
I20260925 05:11:46.869058 26359 raft_consensus.cc:3037] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:46.869134 26626 raft_consensus.cc:3037] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:46.871663 26359 raft_consensus.cc:2454] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e92008215e3445791b89b2f9d5a15e6 in term 1.
I20260925 05:11:46.872076 26426 leader_election.cc:302] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e92008215e3445791b89b2f9d5a15e6, 488bb8a16c1147c0a8c275413fd48496; no voters:
I20260925 05:11:46.872154 26626 raft_consensus.cc:2454] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e92008215e3445791b89b2f9d5a15e6 in term 1.
I20260925 05:11:46.872466 26696 raft_consensus.cc:2781] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:46.872987 26696 raft_consensus.cc:686] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 1 LEADER]: Becoming Leader. State: Replica: 3e92008215e3445791b89b2f9d5a15e6, State: Running, Role: LEADER
I20260925 05:11:46.873531 26696 consensus_queue.cc:237] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } }
I20260925 05:11:46.878007 26215 catalog_manager.cc:5495] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e92008215e3445791b89b2f9d5a15e6 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "3e92008215e3445791b89b2f9d5a15e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e92008215e3445791b89b2f9d5a15e6" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 37289 } health_report { overall_health: HEALTHY } } }
W20260925 05:11:46.898054 26672 tablet.cc:2369] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:46.918860 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.914121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.8.215.2:37289)
0925 05:11:46.914300 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.914304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.914320 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:11:46.914405 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.914409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.914484 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.914611 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:46.914921 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.915610 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.917000 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.917391 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.917484 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.917913 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:46.918385 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:46.918492 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.918563 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.918628 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:11:46.926936 26359 raft_consensus.cc:1261] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 1 FOLLOWER]: Refusing update from remote peer 3e92008215e3445791b89b2f9d5a15e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:46.926939 26626 raft_consensus.cc:1261] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 1 FOLLOWER]: Refusing update from remote peer 3e92008215e3445791b89b2f9d5a15e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:46.927836 26701 consensus_queue.cc:1035] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "488bb8a16c1147c0a8c275413fd48496" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:46.928202 26696 consensus_queue.cc:1035] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc843959d0694e22a87ec369349ab148" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 35661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:46.940254 26705 mvcc.cc:205] Tried to move back new op lower bound from 7333122485962694656 to 7333122485758480384. Current Snapshot: MvccSnapshot[applied={T|T < 7333122485962694656}]
I20260925 05:11:46.954339 26214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:46.954682 26214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:46.958407 26214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ee994c8cb8e4321973d8b0c3eec42f8: Sending DeleteTablet for 3 replicas of tablet f056134e3ec34c98a98f7e7d3c668198
I20260925 05:11:46.959470 26339 tablet_service.cc:1514] Processing DeleteTablet for tablet f056134e3ec34c98a98f7e7d3c668198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:46 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:49916
I20260925 05:11:46.959643 26606 tablet_service.cc:1514] Processing DeleteTablet for tablet f056134e3ec34c98a98f7e7d3c668198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:46 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:49124
I20260925 05:11:46.959815 26473 tablet_service.cc:1514] Processing DeleteTablet for tablet f056134e3ec34c98a98f7e7d3c668198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:46 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:57856
I20260925 05:11:46.960419 26713 tablet_replica.cc:330] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: stopping tablet replica
I20260925 05:11:46.960892 26713 raft_consensus.cc:2229] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:46.961334 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26276
I20260925 05:11:46.961438 26713 raft_consensus.cc:2258] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:46.966171 26713 ts_tablet_manager.cc:1905] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:46.974011 26713 ts_tablet_manager.cc:1918] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:46.974265 26713 log.cc:1196] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/f056134e3ec34c98a98f7e7d3c668198
I20260925 05:11:46.974931 26713 ts_tablet_manager.cc:1939] T f056134e3ec34c98a98f7e7d3c668198 P 488bb8a16c1147c0a8c275413fd48496: Deleting consensus metadata
Sep 25 05:11:46 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:11:46.977131 26199 catalog_manager.cc:4841] TS 488bb8a16c1147c0a8c275413fd48496 (127.8.215.1:36601): tablet f056134e3ec34c98a98f7e7d3c668198 (table test-table [id=cb33554f6b1849a1b3e36bf31a0127da]) successfully deleted
I20260925 05:11:46.978475 26715 tablet_replica.cc:330] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: stopping tablet replica
I20260925 05:11:46.978763 26714 tablet_replica.cc:330] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: stopping tablet replica
I20260925 05:11:46.979048 26715 raft_consensus.cc:2229] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:46.979278 26714 raft_consensus.cc:2229] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:46.979753 26715 raft_consensus.cc:2258] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:46.979853 26714 raft_consensus.cc:2258] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:46.982035 26715 ts_tablet_manager.cc:1905] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:46.982179 26714 ts_tablet_manager.cc:1905] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:46.984254 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:46.962404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.8.215.1:36601)
0925 05:11:46.962568 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:46.962573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:46.964106 (+ 1533us) server_negotiation.cc:407] Connection header received
0925 05:11:46.964298 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:46.964303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:46.964400 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:46.964504 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:46.966131 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.967217 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.969138 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:46.969791 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:46.969895 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:46.976612 (+ 6717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:46.976628 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:46.976645 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:46.976693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:46.980474 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.981418 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.981426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.981430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.981695 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:46.982191 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:46.982196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:46.982199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:46.982715 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:46.983008 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:46.983614 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 05:11:46.983843 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:11:46.985663 26399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:46.990703 26715 ts_tablet_manager.cc:1918] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:46.990957 26715 log.cc:1196] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/f056134e3ec34c98a98f7e7d3c668198
I20260925 05:11:46.991577 26715 ts_tablet_manager.cc:1939] T f056134e3ec34c98a98f7e7d3c668198 P 3e92008215e3445791b89b2f9d5a15e6: Deleting consensus metadata
I20260925 05:11:46.993188 26714 ts_tablet_manager.cc:1918] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:46.993571 26201 catalog_manager.cc:4841] TS 3e92008215e3445791b89b2f9d5a15e6 (127.8.215.2:37289): tablet f056134e3ec34c98a98f7e7d3c668198 (table test-table [id=cb33554f6b1849a1b3e36bf31a0127da]) successfully deleted
I20260925 05:11:46.993657 26714 log.cc:1196] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/f056134e3ec34c98a98f7e7d3c668198
I20260925 05:11:46.994199 26714 ts_tablet_manager.cc:1939] T f056134e3ec34c98a98f7e7d3c668198 P bc843959d0694e22a87ec369349ab148: Deleting consensus metadata
I20260925 05:11:46.995982 26199 catalog_manager.cc:4841] TS bc843959d0694e22a87ec369349ab148 (127.8.215.3:35661): tablet f056134e3ec34c98a98f7e7d3c668198 (table test-table [id=cb33554f6b1849a1b3e36bf31a0127da]) successfully deleted
I20260925 05:11:47.090611 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26276
I20260925 05:11:47.109066 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26410
Sep 25 05:11:47 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:11:47.125413 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:47.109841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.8.215.2:37289)
0925 05:11:47.110371 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:47.110375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:47.110390 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:11:47.110440 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:47.110447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:47.110510 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:47.110628 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:47.111916 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:47.112959 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:47.114470 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:47.114932 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:47.115064 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:47.119177 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:47.119189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:47.119203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:47.119246 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:47.122548 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:47.123225 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:47.123233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:47.123235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:47.123376 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:47.123746 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:47.123751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:47.123754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:47.124100 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:47.124464 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:47.124674 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 05:11:47.124912 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260925 05:11:47.126085 26533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:47.239355 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26410
I20260925 05:11:47.255119 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26543
Sep 25 05:11:47 dist-test-slave-wpdm krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:11:47.270227 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:47.255879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.8.215.3:35661)
0925 05:11:47.256053 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:47.256057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:47.256075 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:11:47.256197 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:47.256201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:47.256271 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:47.256362 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:47.257655 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:47.258363 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:47.259609 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:47.259983 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:47.260069 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:47.264783 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:47.264812 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:47.264836 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:47.264871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:47.267575 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:47.268354 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:47.268361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:47.268363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:47.268518 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:47.268945 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:47.268949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:47.268951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:47.269303 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:47.269544 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:47.269803 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 05:11:47.269979 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:11:47.270910 26666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:47.360126 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26543
I20260925 05:11:47.377342 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26183
I20260925 05:11:47.378445 26245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:47.494539 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26183
2026-09-25T05:11:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T05:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:47Z Disabled control of system clock
I20260925 05:11:47.574200 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43537
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:11:47.903692 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:47.903970 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:47.904053 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:47.911226 26724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:47.911330 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:47.911367 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:47.911423 26724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:47.911465 26724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:47.920645 26724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:47.921921 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:47.923632 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:47.935349 26729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:47.935529 26730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:47.935561 26732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:47.936407 26724 server_base.cc:1034] running on GCE node
I20260925 05:11:47.937284 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:47.938763 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:47.939987 26724 hybrid_clock.cc:648] HybridClock initialized: now 1790313107939943 us; error 38 us; skew 500 ppm
I20260925 05:11:47.940359 26724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:47.946794 26724 webserver.cc:460] Webserver started at http://127.8.215.62:36599/ using document root <none> and password file <none>
I20260925 05:11:47.947543 26724 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:47.947680 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:47.948035 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:47.950968 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "c3babd7dd7bd438aac363b29dda8d3f1"
format_stamp: "Formatted at 2026-09-25 05:11:47 on dist-test-slave-wpdm"
server_key: "33055a031d02393cf05c26ebb2915115"
server_key_iv: "3f66d9454d01c1de1ffe442d93a231df"
server_key_version: "encryptionkey@0"
I20260925 05:11:47.951733 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "c3babd7dd7bd438aac363b29dda8d3f1"
format_stamp: "Formatted at 2026-09-25 05:11:47 on dist-test-slave-wpdm"
server_key: "33055a031d02393cf05c26ebb2915115"
server_key_iv: "3f66d9454d01c1de1ffe442d93a231df"
server_key_version: "encryptionkey@0"
I20260925 05:11:47.956784 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:11:47.960048 26738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:47.961354 26724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:11:47.961532 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "c3babd7dd7bd438aac363b29dda8d3f1"
format_stamp: "Formatted at 2026-09-25 05:11:47 on dist-test-slave-wpdm"
server_key: "33055a031d02393cf05c26ebb2915115"
server_key_iv: "3f66d9454d01c1de1ffe442d93a231df"
server_key_version: "encryptionkey@0"
I20260925 05:11:47.961750 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:47.974464 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:47.975271 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:47.975554 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:47.993858 26724 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43537
I20260925 05:11:47.993880 26789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43537 every 8 connection(s)
I20260925 05:11:47.995165 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:47.997251 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26724
I20260925 05:11:47.997412 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:11:47.997856 9052 external_mini_cluster.cc:1428] Setting key 192f702937281316da760cc198bb7b3f
I20260925 05:11:47.999190 26790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:48.012981 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:47.999758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.8.215.62:43537)
0925 05:11:48.000894 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:48.000908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:48.000960 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:11:48.002713 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:48.002740 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:48.003373 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:48.003807 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:48.004376 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:48.006455 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:48.007849 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:48.008871 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:48.009053 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:48.010107 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:48.010116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:48.010134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:48.010894 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:48.010994 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:48.011037 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:48.011143 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:48.011345 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 05:11:48.011677 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":380,"threads_started":1}
I20260925 05:11:48.014031 26790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1: Bootstrap starting.
I20260925 05:11:48.017189 26790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:48.018173 26790 log.cc:824] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:48.020857 26790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1: No bootstrap required, opened a new log
I20260925 05:11:48.027992 26790 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43537 } }
I20260925 05:11:48.028371 26790 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:48.028527 26790 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3babd7dd7bd438aac363b29dda8d3f1, State: Initialized, Role: FOLLOWER
I20260925 05:11:48.029194 26790 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43537 } }
I20260925 05:11:48.029428 26790 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:48.029558 26790 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:48.029739 26790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:48.031914 26790 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43537 } }
I20260925 05:11:48.032434 26790 leader_election.cc:302] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3babd7dd7bd438aac363b29dda8d3f1; no voters:
I20260925 05:11:48.033178 26790 leader_election.cc:288] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:48.033299 26795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:48.034083 26795 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: c3babd7dd7bd438aac363b29dda8d3f1, State: Running, Role: LEADER
I20260925 05:11:48.034657 26795 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43537 } }
I20260925 05:11:48.035413 26790 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:48.037735 26796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43537 } } }
I20260925 05:11:48.038230 26796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:48.038079 26797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3babd7dd7bd438aac363b29dda8d3f1. Latest consensus state: current_term: 1 leader_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3babd7dd7bd438aac363b29dda8d3f1" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43537 } } }
I20260925 05:11:48.038358 26797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:48.039981 26804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:48.045553 26804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:48.054482 26804 catalog_manager.cc:1347] Generated new cluster ID: 0e350843e52f4aab9e8ddee2795f55d4
I20260925 05:11:48.054600 26804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:48.073453 26804 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:48.074965 26804 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:48.090232 26804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1: Generated new TSK 0
I20260925 05:11:48.091137 26804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:11:48.103664 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43537
--builtin_ntp_servers=127.8.215.20:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:11:48.442749 26814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:48.443008 26814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:48.443094 26814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:48.450230 26814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:48.450326 26814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:48.450430 26814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:11:48.459309 26814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:48.460575 26814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:48.462437 26814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:48.475811 26819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:48.475793 26820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:48.476374 26822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:48.476603 26814 server_base.cc:1034] running on GCE node
I20260925 05:11:48.477209 26814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:48.478152 26814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:48.479378 26814 hybrid_clock.cc:648] HybridClock initialized: now 1790313108479330 us; error 45 us; skew 500 ppm
I20260925 05:11:48.479688 26814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:48.481981 26814 webserver.cc:460] Webserver started at http://127.8.215.1:45725/ using document root <none> and password file <none>
I20260925 05:11:48.482647 26814 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:48.482774 26814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:48.483090 26814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:48.485932 26814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "8d12ec6d276c48dcba825c2db637790c"
format_stamp: "Formatted at 2026-09-25 05:11:48 on dist-test-slave-wpdm"
server_key: "901b873b1ec03c3e6b7f58beabf9d2f1"
server_key_iv: "78a7ea2cf4b805f431dfdb4467094cc6"
server_key_version: "encryptionkey@0"
I20260925 05:11:48.486678 26814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "8d12ec6d276c48dcba825c2db637790c"
format_stamp: "Formatted at 2026-09-25 05:11:48 on dist-test-slave-wpdm"
server_key: "901b873b1ec03c3e6b7f58beabf9d2f1"
server_key_iv: "78a7ea2cf4b805f431dfdb4467094cc6"
server_key_version: "encryptionkey@0"
I20260925 05:11:48.491547 26814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:11:48.494884 26828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:48.496080 26814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 05:11:48.496250 26814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "8d12ec6d276c48dcba825c2db637790c"
format_stamp: "Formatted at 2026-09-25 05:11:48 on dist-test-slave-wpdm"
server_key: "901b873b1ec03c3e6b7f58beabf9d2f1"
server_key_iv: "78a7ea2cf4b805f431dfdb4467094cc6"
server_key_version: "encryptionkey@0"
I20260925 05:11:48.496474 26814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:48.524255 26814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:48.525193 26814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:48.525475 26814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:48.526464 26814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:48.527945 26814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:48.528070 26814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:48.528193 26814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:48.528275 26814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:48.560237 26814 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:38237
I20260925 05:11:48.560321 26940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:38237 every 8 connection(s)
I20260925 05:11:48.561600 26814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:48.563254 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26814
I20260925 05:11:48.563431 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:48.563992 9052 external_mini_cluster.cc:1428] Setting key ba31ad1134ea16144155729481d3f8db
I20260925 05:11:48.566910 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43537
--builtin_ntp_servers=127.8.215.20:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:11:48.575318 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:48.564380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:35685 (local address 127.8.215.62:43537)
0925 05:11:48.564875 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:48.564882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:48.566372 (+ 1490us) server_negotiation.cc:407] Connection header received
0925 05:11:48.568187 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:48.568195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:48.568285 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:48.568399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:48.569480 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:48.570623 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:48.572149 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:48.572940 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:48.573135 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:48.574401 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:48.574409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:48.574417 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:48.574441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:48.574491 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:48.574510 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:48.574615 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:48.574788 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 05:11:48.574899 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
I20260925 05:11:48.576211 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:48.565079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43537 (local address 127.8.215.1:35685)
0925 05:11:48.566209 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:48.566253 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:48.567925 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:48.568605 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:48.568614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:48.568700 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:48.569307 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:48.569326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:48.570759 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:48.570762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:48.572008 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:48.572020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:48.573193 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:48.573198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:48.573393 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:48.574087 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:48.574090 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:48.574132 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:48.574209 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:48.574602 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:48.574609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:48.574615 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:11:48.574752 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 05:11:48.575043 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":370,"threads_started":1}
I20260925 05:11:48.577854 26941 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43537
I20260925 05:11:48.578174 26941 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:48.578920 26941 heartbeater.cc:511] Master 127.8.215.62:43537 requested a full tablet report, sending...
I20260925 05:11:48.580966 26755 ts_manager.cc:194] Registered new tserver with Master: 8d12ec6d276c48dcba825c2db637790c (127.8.215.1:38237)
I20260925 05:11:48.582422 26755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.1:35685
W20260925 05:11:48.893957 26945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:48.894269 26945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:48.894423 26945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:48.901635 26945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:48.901757 26945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:48.901926 26945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:11:48.910887 26945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:48.912186 26945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:48.913971 26945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:48.928107 26951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:48.928056 26950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:48.928197 26953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:48.928761 26945 server_base.cc:1034] running on GCE node
I20260925 05:11:48.929373 26945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:48.930298 26945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:48.931499 26945 hybrid_clock.cc:648] HybridClock initialized: now 1790313108931450 us; error 40 us; skew 500 ppm
I20260925 05:11:48.931799 26945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:48.933995 26945 webserver.cc:460] Webserver started at http://127.8.215.2:38671/ using document root <none> and password file <none>
I20260925 05:11:48.934772 26945 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:48.934916 26945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:48.935230 26945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:48.938266 26945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "989fdd675bdc481db100223cd4cb5af4"
format_stamp: "Formatted at 2026-09-25 05:11:48 on dist-test-slave-wpdm"
server_key: "2ced22036d1fa4eba754d549e8c9f2d9"
server_key_iv: "dfae9b31369cc87e9f4cd914b0b72d5b"
server_key_version: "encryptionkey@0"
I20260925 05:11:48.939144 26945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "989fdd675bdc481db100223cd4cb5af4"
format_stamp: "Formatted at 2026-09-25 05:11:48 on dist-test-slave-wpdm"
server_key: "2ced22036d1fa4eba754d549e8c9f2d9"
server_key_iv: "dfae9b31369cc87e9f4cd914b0b72d5b"
server_key_version: "encryptionkey@0"
I20260925 05:11:48.945192 26945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 05:11:48.948453 26959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:48.949790 26945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:48.949955 26945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "989fdd675bdc481db100223cd4cb5af4"
format_stamp: "Formatted at 2026-09-25 05:11:48 on dist-test-slave-wpdm"
server_key: "2ced22036d1fa4eba754d549e8c9f2d9"
server_key_iv: "dfae9b31369cc87e9f4cd914b0b72d5b"
server_key_version: "encryptionkey@0"
I20260925 05:11:48.950173 26945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:49.004443 26945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:49.005302 26945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:49.005596 26945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:49.006582 26945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:49.008031 26945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:49.008119 26945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:49.008224 26945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:49.008317 26945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:49.041004 26945 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:35481
I20260925 05:11:49.041092 27071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:35481 every 8 connection(s)
I20260925 05:11:49.042369 26945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:49.044222 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 26945
I20260925 05:11:49.044409 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:49.044960 9052 external_mini_cluster.cc:1428] Setting key 06c7082947358ec18d7eff63c2e3d8f3
I20260925 05:11:49.048354 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43537
--builtin_ntp_servers=127.8.215.20:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:11:49.058661 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.045416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:42089 (local address 127.8.215.62:43537)
0925 05:11:49.045638 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.045643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.047437 (+ 1794us) server_negotiation.cc:407] Connection header received
0925 05:11:49.050382 (+ 2945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.050387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.050457 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.050540 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:49.052150 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.052959 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.055646 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.056047 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.056145 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.057916 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:49.057921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:49.057925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:49.057946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:49.057980 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:49.057992 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:49.058065 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.058339 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.058426 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:11:49.059847 27074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.046054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43537 (local address 127.8.215.2:42089)
0925 05:11:49.047252 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.047308 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.050024 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.050838 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.050853 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.050990 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:49.051878 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.051906 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.053130 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.053138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.055492 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.055511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.056291 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.056300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.056662 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.057510 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:49.057517 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:49.057575 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:49.057694 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:49.058091 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:49.058104 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:49.058115 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.058323 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.058813 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":388,"threads_started":1}
I20260925 05:11:49.061458 27072 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43537
I20260925 05:11:49.061784 27072 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:49.062546 27072 heartbeater.cc:511] Master 127.8.215.62:43537 requested a full tablet report, sending...
I20260925 05:11:49.064101 26755 ts_manager.cc:194] Registered new tserver with Master: 989fdd675bdc481db100223cd4cb5af4 (127.8.215.2:35481)
I20260925 05:11:49.065438 26755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.2:42089
W20260925 05:11:49.395634 27075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:49.395958 27075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:49.396059 27075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:49.404349 27075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:49.404464 27075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:49.404582 27075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:11:49.414789 27075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:49.416138 27075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:49.417990 27075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:49.432214 27080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:49.432169 27083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:49.432235 27081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:49.433408 27075 server_base.cc:1034] running on GCE node
I20260925 05:11:49.434047 27075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:49.435276 27075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:49.436493 27075 hybrid_clock.cc:648] HybridClock initialized: now 1790313109436439 us; error 44 us; skew 500 ppm
I20260925 05:11:49.436900 27075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:49.439242 27075 webserver.cc:460] Webserver started at http://127.8.215.3:34875/ using document root <none> and password file <none>
I20260925 05:11:49.439982 27075 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:49.440085 27075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:49.440362 27075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:49.443339 27075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "d56e42015c394f828173114bac3dc125"
format_stamp: "Formatted at 2026-09-25 05:11:49 on dist-test-slave-wpdm"
server_key: "3c2ffe81a5ff8fc3b056e6214603080b"
server_key_iv: "476d424571a2baf7bb6be2a8b67d9946"
server_key_version: "encryptionkey@0"
I20260925 05:11:49.444084 27075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "d56e42015c394f828173114bac3dc125"
format_stamp: "Formatted at 2026-09-25 05:11:49 on dist-test-slave-wpdm"
server_key: "3c2ffe81a5ff8fc3b056e6214603080b"
server_key_iv: "476d424571a2baf7bb6be2a8b67d9946"
server_key_version: "encryptionkey@0"
I20260925 05:11:49.449645 27075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 05:11:49.453325 27089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:49.454813 27075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:11:49.454981 27075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "d56e42015c394f828173114bac3dc125"
format_stamp: "Formatted at 2026-09-25 05:11:49 on dist-test-slave-wpdm"
server_key: "3c2ffe81a5ff8fc3b056e6214603080b"
server_key_iv: "476d424571a2baf7bb6be2a8b67d9946"
server_key_version: "encryptionkey@0"
I20260925 05:11:49.455250 27075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:49.480525 27075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:49.481415 27075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:49.481688 27075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:49.482659 27075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:49.484143 27075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:49.484258 27075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:49.484378 27075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:49.484462 27075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:49.517294 27075 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:44885
I20260925 05:11:49.517382 27201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:44885 every 8 connection(s)
I20260925 05:11:49.518617 27075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:49.519075 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27075
I20260925 05:11:49.519272 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:49.519869 9052 external_mini_cluster.cc:1428] Setting key 1605d4ab8fd5a5e99a7ccc0b6c292221
I20260925 05:11:49.530570 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.521294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:35729 (local address 127.8.215.62:43537)
0925 05:11:49.521488 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.521494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.523112 (+ 1618us) server_negotiation.cc:407] Connection header received
0925 05:11:49.524878 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.524882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.524948 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.525031 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:49.526002 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.526731 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.528192 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.528597 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.528708 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.529953 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:49.529958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:49.529961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:49.529979 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:49.530009 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:49.530021 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:49.530091 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.530267 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.530339 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:11:49.531610 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.521902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43537 (local address 127.8.215.3:35729)
0925 05:11:49.522953 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.523005 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.524602 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.525221 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.525230 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.525320 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:49.525874 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.525890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.526863 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.526866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.528063 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.528075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.528827 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.528832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.529031 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.529650 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:49.529652 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:49.529688 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:49.529764 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:49.530100 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:49.530107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:49.530112 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.530243 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.530512 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":338,"threads_started":1}
I20260925 05:11:49.533016 27202 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43537
I20260925 05:11:49.533336 27202 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:49.534335 27202 heartbeater.cc:511] Master 127.8.215.62:43537 requested a full tablet report, sending...
I20260925 05:11:49.535835 26755 ts_manager.cc:194] Registered new tserver with Master: d56e42015c394f828173114bac3dc125 (127.8.215.3:44885)
I20260925 05:11:49.536782 26755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.3:35729
I20260925 05:11:49.547298 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:49.556805 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.551205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.8.215.62:43537)
0925 05:11:49.551459 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.551465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.551756 (+ 291us) server_negotiation.cc:407] Connection header received
0925 05:11:49.551908 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.551912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.551985 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.552068 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:49.552509 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.553257 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.554591 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.555058 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.555150 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.556127 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:49.556132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:49.556135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:49.556151 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:49.556180 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:49.556191 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:49.556257 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.556380 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.556450 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:11:49.560894 26755 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:49.569883 26755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:49.585289 26941 heartbeater.cc:503] Master 127.8.215.62:43537 was elected leader, sending a full tablet report...
I20260925 05:11:49.591761 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.581696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:35481 (local address 127.0.0.1:48614)
0925 05:11:49.583049 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.583093 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.583264 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.584261 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.584271 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.584309 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:49.584714 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.584743 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.586713 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.586718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.589058 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.589073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.590867 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.590875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.591062 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.591116 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.591247 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.591430 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":945,"threads_started":1}
I20260925 05:11:49.591825 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.581902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.8.215.2:35481)
0925 05:11:49.582706 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.582713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.583186 (+ 473us) server_negotiation.cc:407] Connection header received
0925 05:11:49.583483 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.583492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.583769 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.583967 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:49.584981 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.586546 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.589260 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.590662 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.590822 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.591092 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.591303 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.591472 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":544,"threads_started":1}
I20260925 05:11:49.594748 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.583464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:44885 (local address 127.0.0.1:41698)
0925 05:11:49.583963 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.583981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.584081 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.586055 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.586061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.586084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:49.586486 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.586499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.588508 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.588515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.590730 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.590748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.592984 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.592990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.593193 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.593231 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.594299 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.594437 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260925 05:11:49.595496 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.583833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.8.215.3:44885)
0925 05:11:49.585258 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.585266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.585296 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:11:49.585400 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.585409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.585620 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.585783 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:49.586713 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.588344 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.590949 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.592436 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.592571 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.592850 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.594963 (+ 2113us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.595137 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":1631,"threads_started":1}
I20260925 05:11:49.595839 27007 tablet_service.cc:1467] Processing CreateTablet for tablet 9c526ac230a04e0da0187ef387385d2b (DEFAULT_TABLE table=test-table [id=1876999735f54596b9f5e85ae939d465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:49.596773 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.584638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.8.215.1:38237)
0925 05:11:49.589744 (+ 5106us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.589750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.589772 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:11:49.589833 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.589838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.589991 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.590168 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:49.590694 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.592279 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.594271 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.595708 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.595849 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.596098 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.596262 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.596427 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4957,"thread_start_us":4837,"threads_started":1}
I20260925 05:11:49.597138 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.584011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38237 (local address 127.0.0.1:43418)
0925 05:11:49.584583 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.584601 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.584743 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.590156 (+ 5413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.590161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.590185 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:49.590549 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.590557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.592405 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.592411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.594084 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.594096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.595901 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.595907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.596069 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.596102 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.596198 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.596327 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":845,"threads_started":1}
I20260925 05:11:49.597416 27007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c526ac230a04e0da0187ef387385d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:49.599180 27137 tablet_service.cc:1467] Processing CreateTablet for tablet 9c526ac230a04e0da0187ef387385d2b (DEFAULT_TABLE table=test-table [id=1876999735f54596b9f5e85ae939d465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:49.600112 26876 tablet_service.cc:1467] Processing CreateTablet for tablet 9c526ac230a04e0da0187ef387385d2b (DEFAULT_TABLE table=test-table [id=1876999735f54596b9f5e85ae939d465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:49.600672 27137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c526ac230a04e0da0187ef387385d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:49.601346 26876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c526ac230a04e0da0187ef387385d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:49.612783 27221 tablet_bootstrap.cc:492] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Bootstrap starting.
I20260925 05:11:49.614681 27220 tablet_bootstrap.cc:492] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Bootstrap starting.
I20260925 05:11:49.615556 27222 tablet_bootstrap.cc:492] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Bootstrap starting.
I20260925 05:11:49.617129 27221 tablet_bootstrap.cc:654] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:49.618110 27220 tablet_bootstrap.cc:654] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:49.618436 27221 log.cc:824] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:49.619031 27222 tablet_bootstrap.cc:654] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:49.619423 27220 log.cc:824] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:49.620421 27222 log.cc:824] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:49.622396 27220 tablet_bootstrap.cc:492] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: No bootstrap required, opened a new log
I20260925 05:11:49.622427 27221 tablet_bootstrap.cc:492] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: No bootstrap required, opened a new log
I20260925 05:11:49.622674 27220 ts_tablet_manager.cc:1397] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:11:49.622685 27221 ts_tablet_manager.cc:1397] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 05:11:49.623320 27222 tablet_bootstrap.cc:492] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: No bootstrap required, opened a new log
I20260925 05:11:49.623553 27222 ts_tablet_manager.cc:1397] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:11:49.630373 27221 raft_consensus.cc:348] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.630725 27221 raft_consensus.cc:374] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:49.630851 27221 raft_consensus.cc:729] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989fdd675bdc481db100223cd4cb5af4, State: Initialized, Role: FOLLOWER
I20260925 05:11:49.631446 27221 consensus_queue.cc:260] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.632607 27072 heartbeater.cc:503] Master 127.8.215.62:43537 was elected leader, sending a full tablet report...
I20260925 05:11:49.633147 27221 ts_tablet_manager.cc:1428] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260925 05:11:49.633910 27222 raft_consensus.cc:348] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.634315 27222 raft_consensus.cc:374] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:49.634418 27222 raft_consensus.cc:729] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56e42015c394f828173114bac3dc125, State: Initialized, Role: FOLLOWER
I20260925 05:11:49.634606 27220 raft_consensus.cc:348] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.635058 27220 raft_consensus.cc:374] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:49.635090 27222 consensus_queue.cc:260] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.635164 27220 raft_consensus.cc:729] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d12ec6d276c48dcba825c2db637790c, State: Initialized, Role: FOLLOWER
I20260925 05:11:49.635919 27220 consensus_queue.cc:260] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.640143 27202 heartbeater.cc:503] Master 127.8.215.62:43537 was elected leader, sending a full tablet report...
I20260925 05:11:49.641170 27222 ts_tablet_manager.cc:1428] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Time spent starting tablet: real 0.017s user 0.014s sys 0.005s
I20260925 05:11:49.643831 27220 ts_tablet_manager.cc:1428] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Time spent starting tablet: real 0.021s user 0.009s sys 0.010s
W20260925 05:11:49.770572 27203 tablet.cc:2369] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:11:49.794544 27073 tablet.cc:2369] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:11:49.814100 26942 tablet.cc:2369] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:49.826176 27226 raft_consensus.cc:482] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:49.826526 27226 raft_consensus.cc:504] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.828164 27226 leader_election.cc:288] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56e42015c394f828173114bac3dc125 (127.8.215.3:44885), 8d12ec6d276c48dcba825c2db637790c (127.8.215.1:38237)
I20260925 05:11:49.834885 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.828641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:40909 (local address 127.8.215.3:44885)
0925 05:11:49.828862 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.828866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.829539 (+ 673us) server_negotiation.cc:407] Connection header received
0925 05:11:49.829761 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.829767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.829852 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.829943 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:49.830546 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.831251 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.833194 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.833987 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.834132 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.834233 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.834516 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.834639 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:11:49.834980 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.828579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:44885 (local address 127.8.215.2:40909)
0925 05:11:49.829428 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.829450 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.829588 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.830025 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.830030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.830060 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:49.830401 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.830409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.831393 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.831401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.833013 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.833025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.834176 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.834183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.834348 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.834371 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.834491 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.834617 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":449,"threads_started":1}
I20260925 05:11:49.835376 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.829714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:50601 (local address 127.8.215.1:38237)
0925 05:11:49.829926 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.829932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.830256 (+ 324us) server_negotiation.cc:407] Connection header received
0925 05:11:49.830450 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.830455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.830529 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.830621 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:49.831120 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.831792 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.833687 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.834465 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.834602 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.834711 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.834993 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.835136 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:11:49.835381 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.829557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38237 (local address 127.8.215.2:50601)
0925 05:11:49.830147 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:49.830163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:49.830307 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:49.830635 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:49.830639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:49.830660 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:49.830972 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.830981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.831925 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.831930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.833513 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:49.833525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.834661 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.834666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:49.834841 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.834864 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:11:49.834970 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 05:11:49.835087 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":314,"threads_started":1}
I20260925 05:11:49.836186 27157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c526ac230a04e0da0187ef387385d2b" candidate_uuid: "989fdd675bdc481db100223cd4cb5af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56e42015c394f828173114bac3dc125" is_pre_election: true
I20260925 05:11:49.836230 26896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c526ac230a04e0da0187ef387385d2b" candidate_uuid: "989fdd675bdc481db100223cd4cb5af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d12ec6d276c48dcba825c2db637790c" is_pre_election: true
I20260925 05:11:49.836789 27157 raft_consensus.cc:2454] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 989fdd675bdc481db100223cd4cb5af4 in term 0.
I20260925 05:11:49.836838 26896 raft_consensus.cc:2454] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 989fdd675bdc481db100223cd4cb5af4 in term 0.
I20260925 05:11:49.837463 26961 leader_election.cc:302] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 989fdd675bdc481db100223cd4cb5af4, d56e42015c394f828173114bac3dc125; no voters:
I20260925 05:11:49.837852 27226 raft_consensus.cc:2781] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:49.837997 27226 raft_consensus.cc:482] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:49.838115 27226 raft_consensus.cc:3037] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:49.840368 27226 raft_consensus.cc:504] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.841018 27226 leader_election.cc:288] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [CANDIDATE]: Term 1 election: Requested vote from peers d56e42015c394f828173114bac3dc125 (127.8.215.3:44885), 8d12ec6d276c48dcba825c2db637790c (127.8.215.1:38237)
I20260925 05:11:49.841475 27157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c526ac230a04e0da0187ef387385d2b" candidate_uuid: "989fdd675bdc481db100223cd4cb5af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56e42015c394f828173114bac3dc125"
I20260925 05:11:49.841542 26896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c526ac230a04e0da0187ef387385d2b" candidate_uuid: "989fdd675bdc481db100223cd4cb5af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d12ec6d276c48dcba825c2db637790c"
I20260925 05:11:49.841686 27157 raft_consensus.cc:3037] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:49.841753 26896 raft_consensus.cc:3037] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:49.843837 27157 raft_consensus.cc:2454] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 989fdd675bdc481db100223cd4cb5af4 in term 1.
I20260925 05:11:49.844147 26896 raft_consensus.cc:2454] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 989fdd675bdc481db100223cd4cb5af4 in term 1.
I20260925 05:11:49.844238 26961 leader_election.cc:302] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 989fdd675bdc481db100223cd4cb5af4, d56e42015c394f828173114bac3dc125; no voters:
I20260925 05:11:49.844564 27226 raft_consensus.cc:2781] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:49.845022 27226 raft_consensus.cc:686] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 1 LEADER]: Becoming Leader. State: Replica: 989fdd675bdc481db100223cd4cb5af4, State: Running, Role: LEADER
I20260925 05:11:49.845490 27226 consensus_queue.cc:237] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } }
I20260925 05:11:49.849973 26754 catalog_manager.cc:5495] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 989fdd675bdc481db100223cd4cb5af4 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "989fdd675bdc481db100223cd4cb5af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "989fdd675bdc481db100223cd4cb5af4" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 35481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 } health_report { overall_health: UNKNOWN } } }
I20260925 05:11:49.893199 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.888322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.8.215.2:35481)
0925 05:11:49.888501 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.888505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.888522 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:11:49.888628 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.888632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.888774 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.888897 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:49.889192 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.889870 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.891275 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.891695 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.891788 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.892185 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:49.892699 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:49.892888 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.892931 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.892994 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:11:49.901230 27157 raft_consensus.cc:1261] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 1 FOLLOWER]: Refusing update from remote peer 989fdd675bdc481db100223cd4cb5af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:49.901264 26896 raft_consensus.cc:1261] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 1 FOLLOWER]: Refusing update from remote peer 989fdd675bdc481db100223cd4cb5af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:49.902155 27231 consensus_queue.cc:1035] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d12ec6d276c48dcba825c2db637790c" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:49.902503 27226 consensus_queue.cc:1035] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56e42015c394f828173114bac3dc125" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:49.911327 27234 mvcc.cc:205] Tried to move back new op lower bound from 7333122498145259520 to 7333122497930629120. Current Snapshot: MvccSnapshot[applied={T|T < 7333122498145259520}]
I20260925 05:11:49.912348 27235 mvcc.cc:205] Tried to move back new op lower bound from 7333122498145259520 to 7333122497930629120. Current Snapshot: MvccSnapshot[applied={T|T < 7333122498145259520}]
I20260925 05:11:49.925585 26755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:49.925844 26755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:49.929172 26755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3babd7dd7bd438aac363b29dda8d3f1: Sending DeleteTablet for 3 replicas of tablet 9c526ac230a04e0da0187ef387385d2b
I20260925 05:11:49.930382 27007 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c526ac230a04e0da0187ef387385d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:49 UTC) from {username='slave'} at 127.0.0.1:48614
I20260925 05:11:49.930877 27137 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c526ac230a04e0da0187ef387385d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:49 UTC) from {username='slave'} at 127.0.0.1:41698
I20260925 05:11:49.930920 26876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c526ac230a04e0da0187ef387385d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:49 UTC) from {username='slave'} at 127.0.0.1:43418
I20260925 05:11:49.932085 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26814
I20260925 05:11:49.941859 27244 tablet_replica.cc:330] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: stopping tablet replica
I20260925 05:11:49.942723 27244 raft_consensus.cc:2229] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:49.943688 27244 raft_consensus.cc:2258] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:49.943857 27245 tablet_replica.cc:330] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: stopping tablet replica
I20260925 05:11:49.944437 27245 raft_consensus.cc:2229] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:49.945062 27245 raft_consensus.cc:2258] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:49.946550 27244 ts_tablet_manager.cc:1905] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:49.947005 27245 ts_tablet_manager.cc:1905] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:49.954505 27244 ts_tablet_manager.cc:1918] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:49.954782 27244 log.cc:1196] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/9c526ac230a04e0da0187ef387385d2b
I20260925 05:11:49.955473 27244 ts_tablet_manager.cc:1939] T 9c526ac230a04e0da0187ef387385d2b P 989fdd675bdc481db100223cd4cb5af4: Deleting consensus metadata
I20260925 05:11:49.956990 27246 tablet_replica.cc:330] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: stopping tablet replica
I20260925 05:11:49.957500 27246 raft_consensus.cc:2229] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:49.958204 27246 raft_consensus.cc:2258] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:49.960193 26740 catalog_manager.cc:4841] TS 989fdd675bdc481db100223cd4cb5af4 (127.8.215.2:35481): tablet 9c526ac230a04e0da0187ef387385d2b (table test-table [id=1876999735f54596b9f5e85ae939d465]) successfully deleted
I20260925 05:11:49.961302 27245 ts_tablet_manager.cc:1918] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:49.961539 27245 log.cc:1196] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/9c526ac230a04e0da0187ef387385d2b
I20260925 05:11:49.962678 27245 ts_tablet_manager.cc:1939] T 9c526ac230a04e0da0187ef387385d2b P d56e42015c394f828173114bac3dc125: Deleting consensus metadata
I20260925 05:11:49.963424 27246 ts_tablet_manager.cc:1905] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:49.964547 26740 catalog_manager.cc:4841] TS d56e42015c394f828173114bac3dc125 (127.8.215.3:44885): tablet 9c526ac230a04e0da0187ef387385d2b (table test-table [id=1876999735f54596b9f5e85ae939d465]) successfully deleted
I20260925 05:11:49.967837 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:49.935579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.8.215.1:38237)
0925 05:11:49.935780 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:49.935786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:49.950961 (+ 15175us) server_negotiation.cc:407] Connection header received
0925 05:11:49.951288 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:49.951295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:49.951439 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:49.951519 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:49.952795 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.954038 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.960319 (+ 6281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:49.961104 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:49.961288 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:49.966067 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:49.966080 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:49.966097 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:49.966844 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:49.966913 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:49.966940 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:49.967036 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:49.967298 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:11:49.967420 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:11:49.969292 26936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:50.068343 27246 ts_tablet_manager.cc:1918] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:50.068572 27246 log.cc:1196] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/9c526ac230a04e0da0187ef387385d2b
I20260925 05:11:50.069139 27246 ts_tablet_manager.cc:1939] T 9c526ac230a04e0da0187ef387385d2b P 8d12ec6d276c48dcba825c2db637790c: Deleting consensus metadata
I20260925 05:11:50.071600 26740 catalog_manager.cc:4841] TS 8d12ec6d276c48dcba825c2db637790c (127.8.215.1:38237): tablet 9c526ac230a04e0da0187ef387385d2b (table test-table [id=1876999735f54596b9f5e85ae939d465]) successfully deleted
I20260925 05:11:50.071915 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26814
I20260925 05:11:50.091251 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26945
I20260925 05:11:50.098821 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:50.092123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.8.215.2:35481)
0925 05:11:50.092290 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:50.092298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:50.092350 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:11:50.092573 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:50.092577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:50.092662 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:50.092788 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:50.093188 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:50.094000 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:50.095476 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:50.095896 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:50.096020 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:50.097120 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:50.097129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:50.097144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:50.097895 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:50.097946 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:50.097966 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:50.098077 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:50.098324 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:11:50.098427 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:11:50.099536 27067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:50.297330 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26945
I20260925 05:11:50.315773 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27075
I20260925 05:11:50.322682 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:50.316464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.8.215.3:44885)
0925 05:11:50.316654 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:50.316661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:50.316766 (+ 105us) server_negotiation.cc:407] Connection header received
0925 05:11:50.316969 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:50.316974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:50.317077 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:50.317181 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:50.317567 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:50.318300 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:50.319562 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:50.319964 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:50.320098 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:50.321337 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:50.321347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:50.321362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:50.321919 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:50.321965 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:50.321989 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:50.322113 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:50.322253 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:11:50.322360 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:11:50.323360 27197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:50.414343 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27075
I20260925 05:11:50.432933 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 26724
I20260925 05:11:50.434130 26785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:50.543231 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 26724
2026-09-25T05:11:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3047 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:11:50 dist-test-slave-wpdm krb5kdc[27254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:11:50 dist-test-slave-wpdm krb5kdc[27254](info): set up 2 sockets
Sep 25 05:11:50 dist-test-slave-wpdm krb5kdc[27254](info): commencing operation
krb5kdc: starting...
W20260925 05:11:51.709283 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:11:51 dist-test-slave-wpdm krb5kdc[27254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:11:51.972468 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:33515
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:52.298529 27267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:52.298822 27267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:52.299022 27267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:52.307268 27267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:52.307366 27267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:52.307469 27267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:52.307556 27267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:52.307621 27267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:52.317418 27267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:33515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:52.318714 27267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:52.320073 27267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:52.331858 27273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:52.331928 27275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:52.331897 27272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:52.332587 27267 server_base.cc:1034] running on GCE node
I20260925 05:11:52.333382 27267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:52.334764 27267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:52.335994 27267 hybrid_clock.cc:648] HybridClock initialized: now 1790313112335955 us; error 46 us; skew 500 ppm
I20260925 05:11:52.336333 27267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:52 dist-test-slave-wpdm krb5kdc[27254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:52.339845 27267 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:11:52.341467 27267 webserver.cc:460] Webserver started at http://127.8.215.62:37377/ using document root <none> and password file <none>
I20260925 05:11:52.342180 27267 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:52.342335 27267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:52.342645 27267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:52.345688 27267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "62676e5734d84ae387c1523497b31d7d"
format_stamp: "Formatted at 2026-09-25 05:11:52 on dist-test-slave-wpdm"
server_key: "ff365d3e8216e6063551b4074c766b91"
server_key_iv: "6e4fd826110d6f109eb940a4e949cf10"
server_key_version: "encryptionkey@0"
I20260925 05:11:52.346478 27267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "62676e5734d84ae387c1523497b31d7d"
format_stamp: "Formatted at 2026-09-25 05:11:52 on dist-test-slave-wpdm"
server_key: "ff365d3e8216e6063551b4074c766b91"
server_key_iv: "6e4fd826110d6f109eb940a4e949cf10"
server_key_version: "encryptionkey@0"
I20260925 05:11:52.351671 27267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:11:52.355098 27282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:52.356379 27267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 05:11:52.356564 27267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "62676e5734d84ae387c1523497b31d7d"
format_stamp: "Formatted at 2026-09-25 05:11:52 on dist-test-slave-wpdm"
server_key: "ff365d3e8216e6063551b4074c766b91"
server_key_iv: "6e4fd826110d6f109eb940a4e949cf10"
server_key_version: "encryptionkey@0"
I20260925 05:11:52.356822 27267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:52.385521 27267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:52.390082 27267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:52.390336 27267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:52.408934 27267 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:33515
I20260925 05:11:52.409021 27333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:33515 every 8 connection(s)
I20260925 05:11:52.410298 27267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:52.414481 27334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:52.417218 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27267
I20260925 05:11:52.417413 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:11:52.417872 9052 external_mini_cluster.cc:1428] Setting key d51c7714a83ccc2c1f7b9e2d665c41bb
I20260925 05:11:52.430090 27334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d: Bootstrap starting.
I20260925 05:11:52.434201 27334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d: Neither blocks nor log segments found. Creating new log.
Sep 25 05:11:52 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:52.435436 27334 log.cc:824] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:52.439206 27334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d: No bootstrap required, opened a new log
I20260925 05:11:52.443272 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:52.419731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.8.215.62:33515)
0925 05:11:52.420751 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:52.420770 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:52.420833 (+ 63us) server_negotiation.cc:407] Connection header received
0925 05:11:52.423708 (+ 2875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:52.423740 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:52.424290 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:52.424844 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:52.426107 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:52.428549 (+ 2442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:52.430118 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:52.431306 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:52.431586 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:52.435812 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:52.435821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:52.435837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:52.435881 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:52.439150 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:52.439871 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:52.439885 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:52.439891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:52.440063 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:52.440460 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:52.440464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:52.440469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:52.441059 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:52.441399 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:52.441820 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 05:11:52.442270 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":394,"threads_started":1}
I20260925 05:11:52.449347 27334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62676e5734d84ae387c1523497b31d7d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33515 } }
I20260925 05:11:52.449810 27334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:52.449954 27334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62676e5734d84ae387c1523497b31d7d, State: Initialized, Role: FOLLOWER
I20260925 05:11:52.450702 27334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62676e5734d84ae387c1523497b31d7d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33515 } }
I20260925 05:11:52.451082 27334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:52.451253 27334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:52.451414 27334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:52.454084 27334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62676e5734d84ae387c1523497b31d7d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33515 } }
I20260925 05:11:52.454618 27334 leader_election.cc:302] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62676e5734d84ae387c1523497b31d7d; no voters:
I20260925 05:11:52.455209 27334 leader_election.cc:288] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:52.455310 27339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:52.456262 27339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [term 1 LEADER]: Becoming Leader. State: Replica: 62676e5734d84ae387c1523497b31d7d, State: Running, Role: LEADER
I20260925 05:11:52.457156 27339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62676e5734d84ae387c1523497b31d7d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33515 } }
I20260925 05:11:52.458003 27334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:52.460096 27340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62676e5734d84ae387c1523497b31d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62676e5734d84ae387c1523497b31d7d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33515 } } }
I20260925 05:11:52.460100 27341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62676e5734d84ae387c1523497b31d7d. Latest consensus state: current_term: 1 leader_uuid: "62676e5734d84ae387c1523497b31d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62676e5734d84ae387c1523497b31d7d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 33515 } } }
I20260925 05:11:52.460565 27340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:52.460623 27341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:52.461400 27345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:52.466516 27345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:52.479730 27345 catalog_manager.cc:1347] Generated new cluster ID: b34c490d06914ef8892e8560135f24bb
I20260925 05:11:52.479859 27345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:52.501390 27345 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:52.502661 27345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:52.511404 27345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d: Generated new TSK 0
I20260925 05:11:52.512764 27345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:11:52.582084 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33515
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:52.919974 27360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:52.920269 27360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:52.920441 27360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:52.927812 27360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:52.927953 27360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:52.928159 27360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:11:52.939013 27360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:52.940644 27360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:52.942566 27360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:52.956573 27366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:52.956444 27368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:52.956629 27365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:52.957298 27360 server_base.cc:1034] running on GCE node
I20260925 05:11:52.957898 27360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:52.958932 27360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:52.960155 27360 hybrid_clock.cc:648] HybridClock initialized: now 1790313112960090 us; error 58 us; skew 500 ppm
I20260925 05:11:52.960520 27360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:52 dist-test-slave-wpdm krb5kdc[27254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:52.966998 27360 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:11:52.968411 27360 webserver.cc:460] Webserver started at http://127.8.215.1:33899/ using document root <none> and password file <none>
I20260925 05:11:52.969182 27360 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:52.969336 27360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:52.969642 27360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:52.972539 27360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "7271fed8179149d8882adf85d8e3362a"
format_stamp: "Formatted at 2026-09-25 05:11:52 on dist-test-slave-wpdm"
server_key: "78f87d806528dc7428b8397e5b539819"
server_key_iv: "7dc62643d1ee9ace9dadd77af773a6f9"
server_key_version: "encryptionkey@0"
I20260925 05:11:52.973407 27360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "7271fed8179149d8882adf85d8e3362a"
format_stamp: "Formatted at 2026-09-25 05:11:52 on dist-test-slave-wpdm"
server_key: "78f87d806528dc7428b8397e5b539819"
server_key_iv: "7dc62643d1ee9ace9dadd77af773a6f9"
server_key_version: "encryptionkey@0"
I20260925 05:11:52.978639 27360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:11:52.982148 27375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:52.983459 27360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:11:52.983628 27360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "7271fed8179149d8882adf85d8e3362a"
format_stamp: "Formatted at 2026-09-25 05:11:52 on dist-test-slave-wpdm"
server_key: "78f87d806528dc7428b8397e5b539819"
server_key_iv: "7dc62643d1ee9ace9dadd77af773a6f9"
server_key_version: "encryptionkey@0"
I20260925 05:11:52.983827 27360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:53.002326 27360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:53.006203 27360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:53.006515 27360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:53.007433 27360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:53.008774 27360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:53.008874 27360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:53.009013 27360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:53.009081 27360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:53.041793 27360 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:38251
I20260925 05:11:53.041842 27487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:38251 every 8 connection(s)
I20260925 05:11:53.043107 27360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:53.047482 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27360
I20260925 05:11:53.047659 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:53.048112 9052 external_mini_cluster.cc:1428] Setting key 52d257aa4f02f65e029213547179b233
Sep 25 05:11:53 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:53.063459 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:53.045724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:41603 (local address 127.8.215.62:33515)
0925 05:11:53.046173 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:53.046177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:53.047038 (+ 861us) server_negotiation.cc:407] Connection header received
0925 05:11:53.049092 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:53.049098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:53.049187 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:53.049316 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:53.050871 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.051590 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.053247 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.053647 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.053781 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:53.057256 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:53.057261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:53.057264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:53.057297 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:53.060103 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:53.060886 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:53.060894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:53.060901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:53.061060 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:53.061448 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:53.061455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:53.061462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:53.061865 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:53.062015 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:53.062684 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 05:11:53.062972 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260925 05:11:53.063872 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:53.046182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33515 (local address 127.8.215.1:41603)
0925 05:11:53.046876 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:53.046917 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:53.048833 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:53.049539 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:53.049547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:53.050110 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:53.050695 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:53.050712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.051767 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.051772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:53.053091 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:53.053103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.053902 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.053910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:53.054141 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:53.054817 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:53.054845 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:53.057038 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:53.060278 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:53.060285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:53.060297 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:53.060715 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:53.061153 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:53.061156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:53.061158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:53.061337 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:53.061985 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:53.061991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:53.062295 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 05:11:53.062514 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 05:11:53.062895 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
I20260925 05:11:53.065559 27488 heartbeater.cc:347] Connected to a master server at 127.8.215.62:33515
I20260925 05:11:53.065868 27488 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:53.066572 27488 heartbeater.cc:511] Master 127.8.215.62:33515 requested a full tablet report, sending...
I20260925 05:11:53.068658 27299 ts_manager.cc:194] Registered new tserver with Master: 7271fed8179149d8882adf85d8e3362a (127.8.215.1:38251)
I20260925 05:11:53.070230 27299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:41603
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:11:53.108393 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33515
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:53.431981 27494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:53.432309 27494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:53.432482 27494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:53.439770 27494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:53.439949 27494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:53.440090 27494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:11:53.450515 27494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:53.451720 27494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:53.453011 27494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:53.465586 27500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:53.465623 27499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:53.465550 27502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:53.466387 27494 server_base.cc:1034] running on GCE node
I20260925 05:11:53.467005 27494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:53.467895 27494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:53.469096 27494 hybrid_clock.cc:648] HybridClock initialized: now 1790313113469049 us; error 61 us; skew 500 ppm
I20260925 05:11:53.469426 27494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:53 dist-test-slave-wpdm krb5kdc[27254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:53.472569 27494 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:11:53.474143 27494 webserver.cc:460] Webserver started at http://127.8.215.2:44985/ using document root <none> and password file <none>
I20260925 05:11:53.474843 27494 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:53.474985 27494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:53.475297 27494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:53.478181 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "52faf9420a3c4e389660d9dc817ac053"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-wpdm"
server_key: "5c48a6f52ea2c46c028bd36921ce7caf"
server_key_iv: "bcc37ca8ef588423213ff043a1742bc7"
server_key_version: "encryptionkey@0"
I20260925 05:11:53.478945 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "52faf9420a3c4e389660d9dc817ac053"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-wpdm"
server_key: "5c48a6f52ea2c46c028bd36921ce7caf"
server_key_iv: "bcc37ca8ef588423213ff043a1742bc7"
server_key_version: "encryptionkey@0"
I20260925 05:11:53.483947 27494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:11:53.487530 27509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:53.488956 27494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:11:53.489120 27494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "52faf9420a3c4e389660d9dc817ac053"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-wpdm"
server_key: "5c48a6f52ea2c46c028bd36921ce7caf"
server_key_iv: "bcc37ca8ef588423213ff043a1742bc7"
server_key_version: "encryptionkey@0"
I20260925 05:11:53.489332 27494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:53.513108 27494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:53.517127 27494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:53.517421 27494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:53.518343 27494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:53.519634 27494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:53.519737 27494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:53.519884 27494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:53.519955 27494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:53.552320 27494 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:42945
I20260925 05:11:53.552403 27621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:42945 every 8 connection(s)
I20260925 05:11:53.553689 27494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:53.563004 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27494
I20260925 05:11:53.563249 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:53.563715 9052 external_mini_cluster.cc:1428] Setting key 76628cdf0488ee4628a1f9430be45685
Sep 25 05:11:53 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:53.573793 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:53.556221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:35359 (local address 127.8.215.62:33515)
0925 05:11:53.556413 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:53.556417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:53.557643 (+ 1226us) server_negotiation.cc:407] Connection header received
0925 05:11:53.559580 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:53.559585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:53.559675 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:53.559786 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:53.561378 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.562076 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.563698 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.564194 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.564335 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:53.568073 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:53.568078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:53.568081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:53.568114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:53.570785 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:53.571507 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:53.571511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:53.571513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:53.571630 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:53.572072 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:53.572077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:53.572080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:53.572342 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:53.572495 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:53.573211 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 05:11:53.573431 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:11:53.574800 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:53.556717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33515 (local address 127.8.215.2:35359)
0925 05:11:53.557486 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:53.557532 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:53.559280 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:53.560020 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:53.560028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:53.560588 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:53.561207 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:53.561223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.562228 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.562239 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:53.563561 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:53.563572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.564423 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.564430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:53.564671 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:53.565401 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:53.565422 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:53.567849 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:53.570931 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:53.570939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:53.570956 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:53.571374 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:53.571753 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:53.571757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:53.571761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:53.571954 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:53.572472 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:53.572479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:53.572860 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 05:11:53.573119 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 05:11:53.573658 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":306,"threads_started":1}
I20260925 05:11:53.576295 27622 heartbeater.cc:347] Connected to a master server at 127.8.215.62:33515
I20260925 05:11:53.576619 27622 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:53.577481 27622 heartbeater.cc:511] Master 127.8.215.62:33515 requested a full tablet report, sending...
I20260925 05:11:53.579069 27299 ts_manager.cc:194] Registered new tserver with Master: 52faf9420a3c4e389660d9dc817ac053 (127.8.215.2:42945)
I20260925 05:11:53.580346 27299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:35359
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:11:53.623559 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33515
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:53.955188 27627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:53.955622 27627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:53.955792 27627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:53.965528 27627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:53.965631 27627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:53.965817 27627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:11:53.975724 27627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:53.977303 27627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:53.978700 27627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:53.991910 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:53.991962 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:53.992010 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:53.993070 27627 server_base.cc:1034] running on GCE node
I20260925 05:11:53.993785 27627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:53.994704 27627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:53.995937 27627 hybrid_clock.cc:648] HybridClock initialized: now 1790313113995889 us; error 57 us; skew 500 ppm
I20260925 05:11:53.996265 27627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:53 dist-test-slave-wpdm krb5kdc[27254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:53.999428 27627 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:11:54.000836 27627 webserver.cc:460] Webserver started at http://127.8.215.3:32985/ using document root <none> and password file <none>
I20260925 05:11:54.001475 27627 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:54.001569 27627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:54.001825 27627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:54.004591 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "9a0006def07846cf87a98544beb51917"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-wpdm"
server_key: "8cd4d5c5dbce057b5e9d84373ee4fc1d"
server_key_iv: "c336346952ad1427f4a0ac7507433eb6"
server_key_version: "encryptionkey@0"
I20260925 05:11:54.005400 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "9a0006def07846cf87a98544beb51917"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-wpdm"
server_key: "8cd4d5c5dbce057b5e9d84373ee4fc1d"
server_key_iv: "c336346952ad1427f4a0ac7507433eb6"
server_key_version: "encryptionkey@0"
I20260925 05:11:54.010386 27627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:11:54.013731 27642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.015060 27627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:11:54.015218 27627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "9a0006def07846cf87a98544beb51917"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-wpdm"
server_key: "8cd4d5c5dbce057b5e9d84373ee4fc1d"
server_key_iv: "c336346952ad1427f4a0ac7507433eb6"
server_key_version: "encryptionkey@0"
I20260925 05:11:54.015415 27627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:54.028618 27627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:54.032426 27627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:54.032816 27627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:54.033775 27627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:54.035075 27627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:54.035145 27627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.035315 27627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:54.035368 27627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.068228 27627 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:43309
I20260925 05:11:54.068270 27754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:43309 every 8 connection(s)
I20260925 05:11:54.069640 27627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:54.073159 27488 heartbeater.cc:503] Master 127.8.215.62:33515 was elected leader, sending a full tablet report...
I20260925 05:11:54.078039 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27627
I20260925 05:11:54.078236 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:54.078683 9052 external_mini_cluster.cc:1428] Setting key a6feffeff1e42f5174b7ae1d14ced637
Sep 25 05:11:54 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:54.089710 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.072262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:58599 (local address 127.8.215.62:33515)
0925 05:11:54.072447 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.072451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.073715 (+ 1264us) server_negotiation.cc:407] Connection header received
0925 05:11:54.075647 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.075652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.075754 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.075873 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.077363 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.078104 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.080072 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.080619 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.080779 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.084073 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.084078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.084081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.084110 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.086572 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.087341 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.087346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.087352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.087515 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.087980 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.087984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.087986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.088246 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.088397 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.089113 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.089340 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:11:54.090626 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.072664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:33515 (local address 127.8.215.3:58599)
0925 05:11:54.073547 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.073596 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.075375 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.076065 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.076073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.076583 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:54.077212 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.077230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.078302 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.078323 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.079939 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.079950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.080862 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.080871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.081115 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.081783 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:54.081805 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:54.083882 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:54.086770 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:54.086779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:54.086793 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:54.087218 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:54.087632 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:54.087638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:54.087642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:54.087820 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:54.088383 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:54.088391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:54.088707 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.089016 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.089536 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":346,"threads_started":1}
I20260925 05:11:54.092012 27755 heartbeater.cc:347] Connected to a master server at 127.8.215.62:33515
I20260925 05:11:54.092307 27755 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:54.093163 27755 heartbeater.cc:511] Master 127.8.215.62:33515 requested a full tablet report, sending...
I20260925 05:11:54.094658 27299 ts_manager.cc:194] Registered new tserver with Master: 9a0006def07846cf87a98544beb51917 (127.8.215.3:43309)
I20260925 05:11:54.095453 27299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:58599
I20260925 05:11:54.105994 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:54.122697 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.109974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.8.215.62:33515)
0925 05:11:54.110240 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.110246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.110557 (+ 311us) server_negotiation.cc:407] Connection header received
0925 05:11:54.110687 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.110691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.110751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.110833 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.112208 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.113003 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.114315 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.114732 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.114824 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.117282 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.117287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.117290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.117321 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.120094 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.120803 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.120807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.120809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.120928 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.121312 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.121315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.121317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.121622 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.121834 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.122209 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.122376 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 05:11:54.126468 27299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:54.133440 27299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:54.161865 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.145386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:42945 (local address 127.0.0.1:58446)
0925 05:11:54.147355 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.147416 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.147619 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.149135 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.149149 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.149199 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:54.149811 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.149831 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.153618 (+ 3787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.153626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.158223 (+ 4597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.158245 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.160777 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.160794 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.161076 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.161151 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.161264 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.161474 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"thread_start_us":5188,"threads_started":1}
I20260925 05:11:54.164265 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.146365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38251 (local address 127.0.0.1:55794)
0925 05:11:54.150842 (+ 4477us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.150858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.151011 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.152163 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.152171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.152208 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:54.153394 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.153416 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.156130 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.156139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.159182 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.159204 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.161682 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.161689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.161999 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.162066 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.162186 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.162415 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4372,"thread_start_us":495,"threads_started":1}
I20260925 05:11:54.164297 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.147033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.8.215.1:38251)
0925 05:11:54.150613 (+ 3580us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.150624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.151172 (+ 548us) server_negotiation.cc:407] Connection header received
0925 05:11:54.151288 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.151296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.151565 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.151821 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:54.153695 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.155890 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.159456 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.161437 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.161612 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.161865 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.163454 (+ 1589us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.163949 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3377,"thread_start_us":3180,"threads_started":1}
I20260925 05:11:54.165809 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.145811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.8.215.2:42945)
0925 05:11:54.147944 (+ 2133us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.147958 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.148000 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:11:54.148150 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.148163 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.148489 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.148783 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:54.150076 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.152074 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.158502 (+ 6428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.160531 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.163209 (+ 2678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.165157 (+ 1948us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.165243 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.165455 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1961,"thread_start_us":1865,"threads_started":1}
I20260925 05:11:54.169258 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 70755a0e333041f7aebfca220884cae5 (DEFAULT_TABLE table=test-table [id=28112eb9d60f4f9fa8040e8829a198cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:54.169917 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.150993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:43309 (local address 127.0.0.1:44212)
0925 05:11:54.160569 (+ 9576us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.160589 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.160700 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.162100 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.162105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.162130 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:54.162537 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.162549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.164847 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.164852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.166953 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.166970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.168945 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.168952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.169181 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.169259 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.169337 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.169471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9448,"mutex_wait_us":3,"thread_start_us":9812,"threads_started":1}
I20260925 05:11:54.169718 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 70755a0e333041f7aebfca220884cae5 (DEFAULT_TABLE table=test-table [id=28112eb9d60f4f9fa8040e8829a198cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:54.171105 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.151321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.8.215.3:43309)
0925 05:11:54.161470 (+ 10149us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.161477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.161508 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:11:54.161601 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.161610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.161836 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.162724 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:54.163010 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.164695 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.167197 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.168681 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.170195 (+ 1514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.170483 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.170569 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.170751 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9967,"thread_start_us":9806,"threads_started":1}
I20260925 05:11:54.171267 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70755a0e333041f7aebfca220884cae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:54.171451 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70755a0e333041f7aebfca220884cae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:54.174891 27690 tablet_service.cc:1467] Processing CreateTablet for tablet 70755a0e333041f7aebfca220884cae5 (DEFAULT_TABLE table=test-table [id=28112eb9d60f4f9fa8040e8829a198cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:54.176589 27690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70755a0e333041f7aebfca220884cae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:54.184794 27773 tablet_bootstrap.cc:492] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Bootstrap starting.
I20260925 05:11:54.187346 27774 tablet_bootstrap.cc:492] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Bootstrap starting.
I20260925 05:11:54.189715 27773 tablet_bootstrap.cc:654] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:54.191046 27773 log.cc:824] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:54.191208 27774 tablet_bootstrap.cc:654] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:54.191450 27775 tablet_bootstrap.cc:492] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Bootstrap starting.
I20260925 05:11:54.193114 27774 log.cc:824] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:54.194545 27775 tablet_bootstrap.cc:654] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:54.195781 27775 log.cc:824] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:54.196199 27774 tablet_bootstrap.cc:492] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: No bootstrap required, opened a new log
I20260925 05:11:54.196566 27774 ts_tablet_manager.cc:1397] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 05:11:54.197132 27773 tablet_bootstrap.cc:492] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: No bootstrap required, opened a new log
I20260925 05:11:54.197466 27773 ts_tablet_manager.cc:1397] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260925 05:11:54.199023 27775 tablet_bootstrap.cc:492] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: No bootstrap required, opened a new log
I20260925 05:11:54.199290 27775 ts_tablet_manager.cc:1397] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:11:54.204360 27773 raft_consensus.cc:348] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.204771 27773 raft_consensus.cc:374] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:54.204885 27773 raft_consensus.cc:729] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52faf9420a3c4e389660d9dc817ac053, State: Initialized, Role: FOLLOWER
I20260925 05:11:54.205516 27773 consensus_queue.cc:260] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.206617 27622 heartbeater.cc:503] Master 127.8.215.62:33515 was elected leader, sending a full tablet report...
I20260925 05:11:54.207350 27773 ts_tablet_manager.cc:1428] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260925 05:11:54.208027 27774 raft_consensus.cc:348] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.208477 27774 raft_consensus.cc:374] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:54.208591 27774 raft_consensus.cc:729] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7271fed8179149d8882adf85d8e3362a, State: Initialized, Role: FOLLOWER
I20260925 05:11:54.209250 27774 consensus_queue.cc:260] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.210557 27775 raft_consensus.cc:348] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.211025 27775 raft_consensus.cc:374] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:54.211169 27774 ts_tablet_manager.cc:1428] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260925 05:11:54.211154 27775 raft_consensus.cc:729] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0006def07846cf87a98544beb51917, State: Initialized, Role: FOLLOWER
I20260925 05:11:54.211735 27775 consensus_queue.cc:260] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.213387 27755 heartbeater.cc:503] Master 127.8.215.62:33515 was elected leader, sending a full tablet report...
I20260925 05:11:54.214932 27775 ts_tablet_manager.cc:1428] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
W20260925 05:11:54.295604 27489 tablet.cc:2369] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:11:54.305581 27623 tablet.cc:2369] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:54.311692 27779 raft_consensus.cc:482] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:54.312090 27779 raft_consensus.cc:504] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.313822 27779 leader_election.cc:288] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a0006def07846cf87a98544beb51917 (127.8.215.3:43309), 7271fed8179149d8882adf85d8e3362a (127.8.215.1:38251)
I20260925 05:11:54.320550 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.314218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:36803 (local address 127.8.215.3:43309)
0925 05:11:54.314437 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.314446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.315168 (+ 722us) server_negotiation.cc:407] Connection header received
0925 05:11:54.315384 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.315388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.315447 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.315542 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:54.316154 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.316940 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.318745 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.319555 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.319701 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.319806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.320148 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.320303 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:11:54.320637 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.314210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:43309 (local address 127.8.215.2:36803)
0925 05:11:54.315034 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.315058 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.315225 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.315556 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.315561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.315592 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:54.316006 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.316016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.317056 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.317063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.318566 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.318577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.319754 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.319760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.319993 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.320016 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.320125 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.320293 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":450,"threads_started":1}
I20260925 05:11:54.320776 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.314423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:58591 (local address 127.8.215.1:38251)
0925 05:11:54.314655 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.314663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.315232 (+ 569us) server_negotiation.cc:407] Connection header received
0925 05:11:54.315442 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.315445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.315502 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.315591 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:54.316208 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.316939 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.318821 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.319713 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.319864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.319976 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.320382 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.320502 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:11:54.321147 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.314278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38251 (local address 127.8.215.2:58591)
0925 05:11:54.315131 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.315145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.315303 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.315615 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.315620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.315648 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:54.316064 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.316073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.317061 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.317065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.318655 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.318667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.319986 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.319995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.320216 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.320240 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.320358 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.320480 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":1,"thread_start_us":564,"threads_started":1}
W20260925 05:11:54.321344 27756 tablet.cc:2369] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:54.322062 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70755a0e333041f7aebfca220884cae5" candidate_uuid: "52faf9420a3c4e389660d9dc817ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0006def07846cf87a98544beb51917" is_pre_election: true
I20260925 05:11:54.322307 27443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70755a0e333041f7aebfca220884cae5" candidate_uuid: "52faf9420a3c4e389660d9dc817ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7271fed8179149d8882adf85d8e3362a" is_pre_election: true
I20260925 05:11:54.322602 27710 raft_consensus.cc:2454] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52faf9420a3c4e389660d9dc817ac053 in term 0.
I20260925 05:11:54.322854 27443 raft_consensus.cc:2454] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52faf9420a3c4e389660d9dc817ac053 in term 0.
I20260925 05:11:54.323360 27512 leader_election.cc:302] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52faf9420a3c4e389660d9dc817ac053, 9a0006def07846cf87a98544beb51917; no voters:
I20260925 05:11:54.323757 27779 raft_consensus.cc:2781] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:54.323915 27779 raft_consensus.cc:482] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:54.324023 27779 raft_consensus.cc:3037] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:54.326174 27779 raft_consensus.cc:504] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.326802 27779 leader_election.cc:288] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [CANDIDATE]: Term 1 election: Requested vote from peers 9a0006def07846cf87a98544beb51917 (127.8.215.3:43309), 7271fed8179149d8882adf85d8e3362a (127.8.215.1:38251)
I20260925 05:11:54.327185 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70755a0e333041f7aebfca220884cae5" candidate_uuid: "52faf9420a3c4e389660d9dc817ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0006def07846cf87a98544beb51917"
I20260925 05:11:54.327309 27443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70755a0e333041f7aebfca220884cae5" candidate_uuid: "52faf9420a3c4e389660d9dc817ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7271fed8179149d8882adf85d8e3362a"
I20260925 05:11:54.327368 27710 raft_consensus.cc:3037] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:54.327517 27443 raft_consensus.cc:3037] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:54.329677 27710 raft_consensus.cc:2454] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52faf9420a3c4e389660d9dc817ac053 in term 1.
I20260925 05:11:54.329747 27443 raft_consensus.cc:2454] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52faf9420a3c4e389660d9dc817ac053 in term 1.
I20260925 05:11:54.330072 27512 leader_election.cc:302] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52faf9420a3c4e389660d9dc817ac053, 9a0006def07846cf87a98544beb51917; no voters:
I20260925 05:11:54.330368 27779 raft_consensus.cc:2781] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:54.330742 27779 raft_consensus.cc:686] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 1 LEADER]: Becoming Leader. State: Replica: 52faf9420a3c4e389660d9dc817ac053, State: Running, Role: LEADER
I20260925 05:11:54.331197 27779 consensus_queue.cc:237] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } }
I20260925 05:11:54.335592 27299 catalog_manager.cc:5495] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52faf9420a3c4e389660d9dc817ac053 (127.8.215.2). New cstate: current_term: 1 leader_uuid: "52faf9420a3c4e389660d9dc817ac053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52faf9420a3c4e389660d9dc817ac053" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 42945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 } health_report { overall_health: UNKNOWN } } }
I20260925 05:11:54.351425 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.345867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.8.215.2:42945)
0925 05:11:54.346061 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.346069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.346121 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:11:54.346375 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.346380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.346466 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.346614 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:54.347028 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.347730 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.349239 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.349639 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.349775 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.350346 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:54.350830 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:54.351012 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.351131 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.351188 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:11:54.359511 27710 raft_consensus.cc:1261] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 1 FOLLOWER]: Refusing update from remote peer 52faf9420a3c4e389660d9dc817ac053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:54.359539 27443 raft_consensus.cc:1261] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 1 FOLLOWER]: Refusing update from remote peer 52faf9420a3c4e389660d9dc817ac053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:54.360541 27784 consensus_queue.cc:1035] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0006def07846cf87a98544beb51917" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:54.360980 27779 consensus_queue.cc:1035] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7271fed8179149d8882adf85d8e3362a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:54.372035 27787 mvcc.cc:205] Tried to move back new op lower bound from 7333122516405809152 to 7333122516304130048. Current Snapshot: MvccSnapshot[applied={T|T < 7333122516405809152}]
I20260925 05:11:54.373751 27789 mvcc.cc:205] Tried to move back new op lower bound from 7333122516405809152 to 7333122516304130048. Current Snapshot: MvccSnapshot[applied={T|T < 7333122516405809152}]
I20260925 05:11:54.381563 27788 mvcc.cc:205] Tried to move back new op lower bound from 7333122516405809152 to 7333122516304130048. Current Snapshot: MvccSnapshot[applied={T|T < 7333122516405809152 or (T in {7333122516405809152})}]
I20260925 05:11:54.392691 27299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:54.392987 27299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:54.396550 27299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62676e5734d84ae387c1523497b31d7d: Sending DeleteTablet for 3 replicas of tablet 70755a0e333041f7aebfca220884cae5
I20260925 05:11:54.397791 27690 tablet_service.cc:1514] Processing DeleteTablet for tablet 70755a0e333041f7aebfca220884cae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:44212
I20260925 05:11:54.397828 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 70755a0e333041f7aebfca220884cae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:58446
I20260925 05:11:54.398518 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 70755a0e333041f7aebfca220884cae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:54 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:55794
I20260925 05:11:54.398725 27799 tablet_replica.cc:330] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: stopping tablet replica
I20260925 05:11:54.399164 27799 raft_consensus.cc:2229] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:54.399439 27801 tablet_replica.cc:330] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: stopping tablet replica
I20260925 05:11:54.399677 27799 raft_consensus.cc:2258] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:54.399950 27801 raft_consensus.cc:2229] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:54.400452 27801 raft_consensus.cc:2258] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:54.401947 27799 ts_tablet_manager.cc:1905] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:54.402560 27801 ts_tablet_manager.cc:1905] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:54.404309 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27360
I20260925 05:11:54.408692 27799 ts_tablet_manager.cc:1918] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:54.408854 27799 log.cc:1196] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/70755a0e333041f7aebfca220884cae5
I20260925 05:11:54.409379 27799 ts_tablet_manager.cc:1939] T 70755a0e333041f7aebfca220884cae5 P 9a0006def07846cf87a98544beb51917: Deleting consensus metadata
I20260925 05:11:54.409624 27801 ts_tablet_manager.cc:1918] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:54.409842 27801 log.cc:1196] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/70755a0e333041f7aebfca220884cae5
I20260925 05:11:54.410396 27801 ts_tablet_manager.cc:1939] T 70755a0e333041f7aebfca220884cae5 P 7271fed8179149d8882adf85d8e3362a: Deleting consensus metadata
I20260925 05:11:54.411175 27285 catalog_manager.cc:4841] TS 9a0006def07846cf87a98544beb51917 (127.8.215.3:43309): tablet 70755a0e333041f7aebfca220884cae5 (table test-table [id=28112eb9d60f4f9fa8040e8829a198cc]) successfully deleted
I20260925 05:11:54.412045 27284 catalog_manager.cc:4841] TS 7271fed8179149d8882adf85d8e3362a (127.8.215.1:38251): tablet 70755a0e333041f7aebfca220884cae5 (table test-table [id=28112eb9d60f4f9fa8040e8829a198cc]) successfully deleted
I20260925 05:11:54.420921 27800 tablet_replica.cc:330] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: stopping tablet replica
I20260925 05:11:54.421392 27800 raft_consensus.cc:2229] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:54.422106 27800 raft_consensus.cc:2258] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 05:11:54 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:11:54.424294 27800 ts_tablet_manager.cc:1905] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:54.432057 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.406511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.8.215.1:38251)
0925 05:11:54.406706 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.406713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.413122 (+ 6409us) server_negotiation.cc:407] Connection header received
0925 05:11:54.413265 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.413270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.413345 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.413441 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.415129 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.416274 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.418264 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.418890 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.418988 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.424588 (+ 5600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.424603 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.424620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.424667 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.428450 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.429432 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.429442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.429446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.429612 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.430075 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.430082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.430086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.430562 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.430848 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.431439 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.431668 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:11:54.432256 27800 ts_tablet_manager.cc:1918] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:54.432399 27800 log.cc:1196] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/70755a0e333041f7aebfca220884cae5
I20260925 05:11:54.432960 27800 ts_tablet_manager.cc:1939] T 70755a0e333041f7aebfca220884cae5 P 52faf9420a3c4e389660d9dc817ac053: Deleting consensus metadata
I20260925 05:11:54.433631 27483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:54.435050 27285 catalog_manager.cc:4841] TS 52faf9420a3c4e389660d9dc817ac053 (127.8.215.2:42945): tablet 70755a0e333041f7aebfca220884cae5 (table test-table [id=28112eb9d60f4f9fa8040e8829a198cc]) successfully deleted
I20260925 05:11:54.545819 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27360
I20260925 05:11:54.564903 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27494
Sep 25 05:11:54 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:11:54.581212 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.565640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.8.215.2:42945)
0925 05:11:54.565845 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.565851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.566002 (+ 151us) server_negotiation.cc:407] Connection header received
0925 05:11:54.566212 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.566216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.566277 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.566380 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.567623 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.568464 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.570006 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.570578 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.570727 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.575096 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.575108 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.575123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.575166 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.578338 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.579032 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.579038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.579041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.579216 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.579596 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.579601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.579604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.579969 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.580346 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.580554 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.580803 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 05:11:54.582280 27617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:54.695840 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27494
I20260925 05:11:54.711457 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27627
Sep 25 05:11:54 dist-test-slave-wpdm krb5kdc[27254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:11:54.726137 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.712199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.8.215.3:43309)
0925 05:11:54.712369 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.712374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.712441 (+ 67us) server_negotiation.cc:407] Connection header received
0925 05:11:54.712580 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.712584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.712645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.712789 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.714142 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.714869 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.716122 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.716530 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.716624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.721224 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.721238 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.721254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.721287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.723762 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.724381 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.724388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.724390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.724515 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.724913 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.724916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.724918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.725248 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.725470 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.725701 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.725875 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:11:54.726758 27750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:54.814805 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27627
I20260925 05:11:54.832551 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27267
I20260925 05:11:54.833721 27329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:54.949476 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27267
2026-09-25T05:11:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4407 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T05:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:55Z Disabled control of system clock
I20260925 05:11:55.032455 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43265
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:11:55.372285 27810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:55.372571 27810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:55.372666 27810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:55.380177 27810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:55.380290 27810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:55.380326 27810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:55.380383 27810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:55.380427 27810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:55.389778 27810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:55.390982 27810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:55.392683 27810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:55.404620 27815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:55.404590 27816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:55.404743 27818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:55.405463 27810 server_base.cc:1034] running on GCE node
I20260925 05:11:55.406229 27810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:55.407606 27810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:55.408833 27810 hybrid_clock.cc:648] HybridClock initialized: now 1790313115408767 us; error 44 us; skew 500 ppm
I20260925 05:11:55.409184 27810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:55.411273 27810 webserver.cc:460] Webserver started at http://127.8.215.62:33013/ using document root <none> and password file <none>
I20260925 05:11:55.411924 27810 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:55.412053 27810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:55.412369 27810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:55.415199 27810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "b2eb17b489db4781b1a0767d16ee89f0"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-wpdm"
server_key: "e22e807418fe23426456b37cd92708b7"
server_key_iv: "89f89d84a91b4a18a0c6e6f1efde9d9c"
server_key_version: "encryptionkey@0"
I20260925 05:11:55.415979 27810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "b2eb17b489db4781b1a0767d16ee89f0"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-wpdm"
server_key: "e22e807418fe23426456b37cd92708b7"
server_key_iv: "89f89d84a91b4a18a0c6e6f1efde9d9c"
server_key_version: "encryptionkey@0"
I20260925 05:11:55.420862 27810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 05:11:55.424101 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:55.425477 27810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:55.425659 27810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "b2eb17b489db4781b1a0767d16ee89f0"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-wpdm"
server_key: "e22e807418fe23426456b37cd92708b7"
server_key_iv: "89f89d84a91b4a18a0c6e6f1efde9d9c"
server_key_version: "encryptionkey@0"
I20260925 05:11:55.425865 27810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:55.443140 27810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:55.443924 27810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:55.444227 27810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:55.462635 27810 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43265
I20260925 05:11:55.462653 27875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43265 every 8 connection(s)
I20260925 05:11:55.464156 27810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:55.466769 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27810
I20260925 05:11:55.466952 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:11:55.467489 9052 external_mini_cluster.cc:1428] Setting key c804aa5e32d409684e7c9956f30d229d
I20260925 05:11:55.468544 27876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:55.481108 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.469427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.8.215.62:43265)
0925 05:11:55.470380 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.470390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.470431 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:11:55.471906 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.471926 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.472376 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.472712 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:55.473302 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.474651 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.476649 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.477372 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.477513 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.478782 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:55.478792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:55.478806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:55.479356 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:55.479428 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:55.479450 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:55.479549 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.479793 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.480142 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":370,"threads_started":1}
I20260925 05:11:55.482370 27876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0: Bootstrap starting.
I20260925 05:11:55.485526 27876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:55.486475 27876 log.cc:824] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:55.489041 27876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0: No bootstrap required, opened a new log
I20260925 05:11:55.496398 27876 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eb17b489db4781b1a0767d16ee89f0" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43265 } }
I20260925 05:11:55.496752 27876 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:55.496877 27876 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2eb17b489db4781b1a0767d16ee89f0, State: Initialized, Role: FOLLOWER
I20260925 05:11:55.497476 27876 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eb17b489db4781b1a0767d16ee89f0" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43265 } }
I20260925 05:11:55.497699 27876 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:55.497822 27876 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:55.497987 27876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:55.500306 27876 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eb17b489db4781b1a0767d16ee89f0" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43265 } }
I20260925 05:11:55.500845 27876 leader_election.cc:302] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2eb17b489db4781b1a0767d16ee89f0; no voters:
I20260925 05:11:55.501391 27876 leader_election.cc:288] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:55.501521 27881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:55.502255 27881 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [term 1 LEADER]: Becoming Leader. State: Replica: b2eb17b489db4781b1a0767d16ee89f0, State: Running, Role: LEADER
I20260925 05:11:55.502720 27881 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eb17b489db4781b1a0767d16ee89f0" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43265 } }
I20260925 05:11:55.503432 27876 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:55.505854 27883 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2eb17b489db4781b1a0767d16ee89f0. Latest consensus state: current_term: 1 leader_uuid: "b2eb17b489db4781b1a0767d16ee89f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eb17b489db4781b1a0767d16ee89f0" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43265 } } }
I20260925 05:11:55.506208 27883 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:55.507133 27882 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2eb17b489db4781b1a0767d16ee89f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eb17b489db4781b1a0767d16ee89f0" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43265 } } }
I20260925 05:11:55.508569 27882 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:55.509373 27888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:55.514593 27888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:55.523032 27888 catalog_manager.cc:1347] Generated new cluster ID: 36be7f1a315c4abba6a2bf0542fa35af
I20260925 05:11:55.523146 27888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:55.547400 27888 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:55.548705 27888 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:55.562057 27888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0: Generated new TSK 0
I20260925 05:11:55.562928 27888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:11:55.568898 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43265
--builtin_ntp_servers=127.8.215.20:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:11:55.906742 27900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:55.907011 27900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:55.907152 27900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:55.914683 27900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:55.914778 27900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:55.914958 27900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:11:55.924211 27900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:55.925603 27900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:55.927325 27900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:55.941162 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:55.941135 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:55.941219 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:55.942443 27900 server_base.cc:1034] running on GCE node
I20260925 05:11:55.943030 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:55.943943 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:55.945204 27900 hybrid_clock.cc:648] HybridClock initialized: now 1790313115945126 us; error 50 us; skew 500 ppm
I20260925 05:11:55.945565 27900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:55.948024 27900 webserver.cc:460] Webserver started at http://127.8.215.1:39531/ using document root <none> and password file <none>
I20260925 05:11:55.948669 27900 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:55.948830 27900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:55.949111 27900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:55.951855 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "79e2ec9af1c448dda97fc79b9878261a"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-wpdm"
server_key: "49edb56477ddc7d8f47765db7edf7679"
server_key_iv: "f58bb6b9c3daac3bdd0b78880a3f413b"
server_key_version: "encryptionkey@0"
I20260925 05:11:55.952585 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "79e2ec9af1c448dda97fc79b9878261a"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-wpdm"
server_key: "49edb56477ddc7d8f47765db7edf7679"
server_key_iv: "f58bb6b9c3daac3bdd0b78880a3f413b"
server_key_version: "encryptionkey@0"
I20260925 05:11:55.957623 27900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 05:11:55.960856 27914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:55.962133 27900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:11:55.962261 27900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "79e2ec9af1c448dda97fc79b9878261a"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-wpdm"
server_key: "49edb56477ddc7d8f47765db7edf7679"
server_key_iv: "f58bb6b9c3daac3bdd0b78880a3f413b"
server_key_version: "encryptionkey@0"
I20260925 05:11:55.962431 27900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:55.977649 27900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:55.978423 27900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:55.978652 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:55.979604 27900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:55.981026 27900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:55.981124 27900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:55.981206 27900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:55.981261 27900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.014256 27900 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:38879
I20260925 05:11:56.014304 28026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:38879 every 8 connection(s)
I20260925 05:11:56.015553 27900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:56.017098 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 27900
I20260925 05:11:56.017275 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:56.017853 9052 external_mini_cluster.cc:1428] Setting key 63c79f4e5df7edf2de5d4ff154f55c53
I20260925 05:11:56.021097 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43265
--builtin_ntp_servers=127.8.215.20:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:11:56.031030 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.018101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:39533 (local address 127.8.215.62:43265)
0925 05:11:56.018837 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:56.018843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:56.020139 (+ 1296us) server_negotiation.cc:407] Connection header received
0925 05:11:56.023019 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:56.023023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:56.023132 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:56.023229 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:56.024703 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.025504 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.028007 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.028456 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.028559 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:56.030259 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:56.030264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:56.030267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:56.030285 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:56.030325 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:56.030337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:56.030446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:56.030664 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 05:11:56.030750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":483,"threads_started":1}
I20260925 05:11:56.032575 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.018686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43265 (local address 127.8.215.1:39533)
0925 05:11:56.019940 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:56.020016 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:56.022685 (+ 2669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:56.023503 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:56.023517 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:56.023645 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:56.024498 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:56.024525 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.025655 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.025663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:56.027848 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:56.027867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.028681 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.028690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:56.029091 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:56.029883 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:56.029889 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:56.029944 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:56.030046 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:56.030437 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:56.030450 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:56.030462 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:11:56.030645 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 05:11:56.031117 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
I20260925 05:11:56.034193 28027 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43265
I20260925 05:11:56.034518 28027 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:56.035291 28027 heartbeater.cc:511] Master 127.8.215.62:43265 requested a full tablet report, sending...
I20260925 05:11:56.037496 27841 ts_manager.cc:194] Registered new tserver with Master: 79e2ec9af1c448dda97fc79b9878261a (127.8.215.1:38879)
I20260925 05:11:56.038983 27841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.1:39533
W20260925 05:11:56.348611 28031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:56.349174 28031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:56.349426 28031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:56.360867 28031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:56.361016 28031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:56.361248 28031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:11:56.376252 28031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:56.378201 28031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:56.380692 28031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:56.397105 28039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:56.397128 28037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:56.397428 28036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:56.397815 28031 server_base.cc:1034] running on GCE node
I20260925 05:11:56.398448 28031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:56.399427 28031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:56.400628 28031 hybrid_clock.cc:648] HybridClock initialized: now 1790313116400555 us; error 73 us; skew 500 ppm
I20260925 05:11:56.401005 28031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:56.403352 28031 webserver.cc:460] Webserver started at http://127.8.215.2:43795/ using document root <none> and password file <none>
I20260925 05:11:56.404116 28031 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:56.404260 28031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:56.404599 28031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:56.407472 28031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "a430ef1e1448423ba4179d4ba6139530"
format_stamp: "Formatted at 2026-09-25 05:11:56 on dist-test-slave-wpdm"
server_key: "353f9c5e9aed15a39e4ba944a666d19f"
server_key_iv: "bf847a091766eb087e777880b165bbf7"
server_key_version: "encryptionkey@0"
I20260925 05:11:56.408255 28031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "a430ef1e1448423ba4179d4ba6139530"
format_stamp: "Formatted at 2026-09-25 05:11:56 on dist-test-slave-wpdm"
server_key: "353f9c5e9aed15a39e4ba944a666d19f"
server_key_iv: "bf847a091766eb087e777880b165bbf7"
server_key_version: "encryptionkey@0"
I20260925 05:11:56.413561 28031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:11:56.417147 28045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.418510 28031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:11:56.418692 28031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "a430ef1e1448423ba4179d4ba6139530"
format_stamp: "Formatted at 2026-09-25 05:11:56 on dist-test-slave-wpdm"
server_key: "353f9c5e9aed15a39e4ba944a666d19f"
server_key_iv: "bf847a091766eb087e777880b165bbf7"
server_key_version: "encryptionkey@0"
I20260925 05:11:56.418928 28031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:56.443142 28031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:56.444077 28031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:56.444373 28031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:56.445477 28031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:56.446947 28031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:56.447067 28031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.447201 28031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:56.447299 28031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.484608 28031 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:36773
I20260925 05:11:56.484673 28157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:36773 every 8 connection(s)
I20260925 05:11:56.486231 28031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:56.490602 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28031
I20260925 05:11:56.490806 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:56.491441 9052 external_mini_cluster.cc:1428] Setting key 1f15b674b0c73f89b461836e8c4cfbb5
I20260925 05:11:56.494741 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43265
--builtin_ntp_servers=127.8.215.20:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:11:56.501523 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.488998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:48977 (local address 127.8.215.62:43265)
0925 05:11:56.489207 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:56.489212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:56.491143 (+ 1931us) server_negotiation.cc:407] Connection header received
0925 05:11:56.493623 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:56.493628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:56.493687 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:56.493780 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:56.495305 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.496001 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.498590 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.498963 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.499055 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:56.500777 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:56.500789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:56.500794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:56.500813 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:56.500845 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:56.500858 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:56.500934 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:56.501226 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 05:11:56.501306 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:11:56.503168 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.489635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43265 (local address 127.8.215.2:48977)
0925 05:11:56.490933 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:56.490993 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:56.493323 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:56.494092 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:56.494104 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:56.494230 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:56.495124 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:56.495157 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.496159 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.496168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:56.498444 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:56.498462 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.499204 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.499214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:56.499578 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:56.500355 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:56.500361 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:56.500418 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:56.500530 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:56.500961 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:56.500975 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:56.500986 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:11:56.501207 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 05:11:56.501659 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":440,"threads_started":1}
I20260925 05:11:56.504689 28158 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43265
I20260925 05:11:56.505064 28158 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:56.505836 28158 heartbeater.cc:511] Master 127.8.215.62:43265 requested a full tablet report, sending...
I20260925 05:11:56.507380 27841 ts_manager.cc:194] Registered new tserver with Master: a430ef1e1448423ba4179d4ba6139530 (127.8.215.2:36773)
I20260925 05:11:56.508641 27841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.2:48977
W20260925 05:11:56.832432 28161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:56.832707 28161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:56.832813 28161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:56.839932 28161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:56.840031 28161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:56.840142 28161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:11:56.849187 28161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:43265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:56.850538 28161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:56.852281 28161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:56.866580 28166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:56.866452 28167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:56.866596 28169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:56.870299 28161 server_base.cc:1034] running on GCE node
I20260925 05:11:56.870900 28161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:56.871855 28161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:56.873095 28161 hybrid_clock.cc:648] HybridClock initialized: now 1790313116873037 us; error 45 us; skew 500 ppm
I20260925 05:11:56.873446 28161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:11:56.875684 28161 webserver.cc:460] Webserver started at http://127.8.215.3:37983/ using document root <none> and password file <none>
I20260925 05:11:56.876365 28161 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:56.876497 28161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:56.876880 28161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:56.879746 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0"
format_stamp: "Formatted at 2026-09-25 05:11:56 on dist-test-slave-wpdm"
server_key: "38eb0be66e7fddb7993396f0b19b491d"
server_key_iv: "e967b1dc4f56cb47fe3c73f1b2907be2"
server_key_version: "encryptionkey@0"
I20260925 05:11:56.880544 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0"
format_stamp: "Formatted at 2026-09-25 05:11:56 on dist-test-slave-wpdm"
server_key: "38eb0be66e7fddb7993396f0b19b491d"
server_key_iv: "e967b1dc4f56cb47fe3c73f1b2907be2"
server_key_version: "encryptionkey@0"
I20260925 05:11:56.885684 28161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 05:11:56.889009 28175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.890434 28161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:11:56.890630 28161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0"
format_stamp: "Formatted at 2026-09-25 05:11:56 on dist-test-slave-wpdm"
server_key: "38eb0be66e7fddb7993396f0b19b491d"
server_key_iv: "e967b1dc4f56cb47fe3c73f1b2907be2"
server_key_version: "encryptionkey@0"
I20260925 05:11:56.890856 28161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:56.915745 28161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:56.916692 28161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:56.917019 28161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:56.918255 28161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:56.919844 28161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:56.919996 28161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.920147 28161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:56.920248 28161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:56.961498 28161 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:33651
I20260925 05:11:56.961557 28287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:33651 every 8 connection(s)
I20260925 05:11:56.962795 28161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:56.964576 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28161
I20260925 05:11:56.964797 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:56.965372 9052 external_mini_cluster.cc:1428] Setting key 12c121cc4455f79db319bcda9bb16337
I20260925 05:11:56.975478 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.965448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:38755 (local address 127.8.215.62:43265)
0925 05:11:56.965643 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:56.965650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:56.967394 (+ 1744us) server_negotiation.cc:407] Connection header received
0925 05:11:56.969541 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:56.969546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:56.969640 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:56.969728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:56.970710 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.971451 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.973000 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.973424 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.973526 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:56.974796 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:56.974801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:56.974804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:56.974821 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:56.974850 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:56.974860 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:56.974928 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:56.975169 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 05:11:56.975250 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:11:56.976423 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.966093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:43265 (local address 127.8.215.3:38755)
0925 05:11:56.967239 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:56.967280 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:56.969302 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:56.969919 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:56.969927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:56.970016 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:56.970565 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:56.970580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.971575 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.971579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:56.972872 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:56.972884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.973611 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:56.973615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:56.973827 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:56.974515 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:56.974518 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:56.974562 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:56.974641 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:56.974940 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:56.974948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:56.974953 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:11:56.975092 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 05:11:56.975385 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":325,"threads_started":1}
I20260925 05:11:56.977806 28288 heartbeater.cc:347] Connected to a master server at 127.8.215.62:43265
I20260925 05:11:56.978097 28288 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:56.978860 28288 heartbeater.cc:511] Master 127.8.215.62:43265 requested a full tablet report, sending...
I20260925 05:11:56.980572 27841 ts_manager.cc:194] Registered new tserver with Master: 263d9c7e012d41bd91ce9e0f5f91d0f0 (127.8.215.3:33651)
I20260925 05:11:56.981418 27841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.3:38755
I20260925 05:11:56.993309 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:57.003408 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:56.997419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.8.215.62:43265)
0925 05:11:56.997610 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:56.997618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:56.997947 (+ 329us) server_negotiation.cc:407] Connection header received
0925 05:11:56.998133 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:56.998139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:56.998231 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:56.998334 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:56.998786 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:56.999489 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.000943 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.001390 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.001534 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.002650 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:57.002660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:57.002663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:57.002680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:57.002710 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:57.002721 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:57.002865 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.003083 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.003139 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:11:57.007300 27841 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:57.014273 27841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:57.042207 28027 heartbeater.cc:503] Master 127.8.215.62:43265 was elected leader, sending a full tablet report...
I20260925 05:11:57.044998 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.026868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:33651 (local address 127.0.0.1:44896)
0925 05:11:57.030456 (+ 3588us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.030470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:57.030609 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:57.032234 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:57.032245 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:57.032286 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:57.032880 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.032897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.037097 (+ 4200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.037106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.041880 (+ 4774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.041903 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.044009 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.044017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.044234 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.044308 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 05:11:57.044421 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:11:57.044632 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3492,"thread_start_us":3364,"threads_started":1}
I20260925 05:11:57.045614 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.027476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.8.215.3:33651)
0925 05:11:57.030293 (+ 2817us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.030303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.030572 (+ 269us) server_negotiation.cc:407] Connection header received
0925 05:11:57.031233 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.031241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.031520 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.031825 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:57.033160 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.036892 (+ 3732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.042138 (+ 5246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.043632 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.043804 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.044867 (+ 1063us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.044957 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.045140 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2574,"thread_start_us":2434,"threads_started":1}
I20260925 05:11:57.049645 28223 tablet_service.cc:1467] Processing CreateTablet for tablet 5a327acdf91c4bc08cfa569adec57a26 (DEFAULT_TABLE table=test-table [id=3f7f896fdc0e4c2295ffbba7f01893d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:57.050880 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.026524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51298)
0925 05:11:57.029204 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.029263 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:57.029475 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:57.043478 (+ 14003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:57.043486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:57.043517 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:57.043888 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.043902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.046138 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.046144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.048174 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.048187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.050033 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.050040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.050254 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.050286 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:11:57.050369 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:11:57.050508 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":19,"thread_start_us":623,"threads_started":1}
I20260925 05:11:57.051276 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.026778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.8.215.1:38879)
0925 05:11:57.042095 (+ 15317us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.042108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.042153 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:11:57.042311 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.042321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.042651 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.044100 (+ 1449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:57.044343 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.046004 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.048417 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.049836 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.049972 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.050358 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.050436 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.050688 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15041,"thread_start_us":16414,"threads_started":1}
I20260925 05:11:57.051434 28223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a327acdf91c4bc08cfa569adec57a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:57.054277 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.030863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36773 (local address 127.0.0.1:34144)
0925 05:11:57.043298 (+ 12435us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.043317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:57.043445 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:57.047684 (+ 4239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:57.047689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:57.047716 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:57.048087 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.048101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.049500 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.049505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.052025 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.052037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.053381 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.053387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.053553 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.053605 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 05:11:57.053753 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 05:11:57.053912 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12302,"mutex_wait_us":1,"thread_start_us":12150,"threads_started":1}
I20260925 05:11:57.054374 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.030958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.8.215.2:36773)
0925 05:11:57.047090 (+ 16132us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.047099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.047131 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:11:57.047228 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.047235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.047471 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.047676 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:57.048297 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.049363 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.052200 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.053180 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.053321 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.053555 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.053829 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.054003 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15952,"thread_start_us":16907,"threads_started":1}
I20260925 05:11:57.056033 27962 tablet_service.cc:1467] Processing CreateTablet for tablet 5a327acdf91c4bc08cfa569adec57a26 (DEFAULT_TABLE table=test-table [id=3f7f896fdc0e4c2295ffbba7f01893d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:57.057375 27962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a327acdf91c4bc08cfa569adec57a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:57.060496 28093 tablet_service.cc:1467] Processing CreateTablet for tablet 5a327acdf91c4bc08cfa569adec57a26 (DEFAULT_TABLE table=test-table [id=3f7f896fdc0e4c2295ffbba7f01893d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:57.061856 28093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a327acdf91c4bc08cfa569adec57a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:57.063611 28306 tablet_bootstrap.cc:492] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Bootstrap starting.
I20260925 05:11:57.067296 28306 tablet_bootstrap.cc:654] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:57.068414 28306 log.cc:824] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:57.070955 28306 tablet_bootstrap.cc:492] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: No bootstrap required, opened a new log
I20260925 05:11:57.071236 28306 ts_tablet_manager.cc:1397] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:11:57.072758 28308 tablet_bootstrap.cc:492] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: Bootstrap starting.
I20260925 05:11:57.076186 28309 tablet_bootstrap.cc:492] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Bootstrap starting.
I20260925 05:11:57.076567 28308 tablet_bootstrap.cc:654] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:57.077895 28308 log.cc:824] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:57.079434 28309 tablet_bootstrap.cc:654] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:57.080394 28309 log.cc:824] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:57.082231 28306 raft_consensus.cc:348] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.082619 28306 raft_consensus.cc:374] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:57.082713 28306 raft_consensus.cc:729] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263d9c7e012d41bd91ce9e0f5f91d0f0, State: Initialized, Role: FOLLOWER
I20260925 05:11:57.083302 28308 tablet_bootstrap.cc:492] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: No bootstrap required, opened a new log
I20260925 05:11:57.083298 28309 tablet_bootstrap.cc:492] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: No bootstrap required, opened a new log
I20260925 05:11:57.083339 28306 consensus_queue.cc:260] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.083573 28308 ts_tablet_manager.cc:1397] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 05:11:57.083573 28309 ts_tablet_manager.cc:1397] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:11:57.084427 28288 heartbeater.cc:503] Master 127.8.215.62:43265 was elected leader, sending a full tablet report...
I20260925 05:11:57.086295 28306 ts_tablet_manager.cc:1428] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 05:11:57.094415 28309 raft_consensus.cc:348] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.094417 28308 raft_consensus.cc:348] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.094894 28308 raft_consensus.cc:374] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:57.094892 28309 raft_consensus.cc:374] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:57.095005 28308 raft_consensus.cc:729] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e2ec9af1c448dda97fc79b9878261a, State: Initialized, Role: FOLLOWER
I20260925 05:11:57.094997 28309 raft_consensus.cc:729] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a430ef1e1448423ba4179d4ba6139530, State: Initialized, Role: FOLLOWER
I20260925 05:11:57.095647 28308 consensus_queue.cc:260] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.095660 28309 consensus_queue.cc:260] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.096771 28158 heartbeater.cc:503] Master 127.8.215.62:43265 was elected leader, sending a full tablet report...
I20260925 05:11:57.097550 28309 ts_tablet_manager.cc:1428] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 05:11:57.097759 28308 ts_tablet_manager.cc:1428] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260925 05:11:57.214493 28289 tablet.cc:2369] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:11:57.238322 28159 tablet.cc:2369] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:11:57.267942 28028 tablet.cc:2369] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:57.303500 28312 raft_consensus.cc:482] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:57.303925 28312 raft_consensus.cc:504] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.305894 28312 leader_election.cc:288] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e2ec9af1c448dda97fc79b9878261a (127.8.215.1:38879), a430ef1e1448423ba4179d4ba6139530 (127.8.215.2:36773)
I20260925 05:11:57.312958 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.306379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:45227 (local address 127.8.215.1:38879)
0925 05:11:57.306602 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.306609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.306634 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:11:57.306885 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.306892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.307002 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.307128 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:57.307710 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.308843 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.310741 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.312048 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.312152 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.312231 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.312598 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.312698 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:11:57.313016 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.306182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.8.215.3:45227)
0925 05:11:57.306501 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.306524 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:57.306683 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:57.307187 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:57.307191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:57.307218 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:57.307514 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.307525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.308984 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.308990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.310588 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.310601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.312206 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.312212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.312409 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.312444 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:11:57.312542 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:11:57.312679 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"spinlock_wait_cycles":38144}
I20260925 05:11:57.314103 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.306379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:52827 (local address 127.8.215.2:36773)
0925 05:11:57.306590 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.306598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.307299 (+ 701us) server_negotiation.cc:407] Connection header received
0925 05:11:57.307489 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.307495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.307587 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.307686 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:57.308371 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.309368 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.311859 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.312932 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.313062 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.313153 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.313691 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.313853 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:11:57.314128 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.306255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36773 (local address 127.8.215.3:52827)
0925 05:11:57.307177 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.307199 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:57.307347 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:57.307716 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:57.307720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:57.307749 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:57.308219 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.308233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.309520 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.309524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.311646 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:57.311665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.313334 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.313341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:57.313527 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.313564 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:11:57.313656 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:11:57.313810 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":547,"threads_started":1}
I20260925 05:11:57.314774 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a327acdf91c4bc08cfa569adec57a26" candidate_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e2ec9af1c448dda97fc79b9878261a" is_pre_election: true
I20260925 05:11:57.315384 27982 raft_consensus.cc:2454] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 263d9c7e012d41bd91ce9e0f5f91d0f0 in term 0.
I20260925 05:11:57.315357 28113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a327acdf91c4bc08cfa569adec57a26" candidate_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a430ef1e1448423ba4179d4ba6139530" is_pre_election: true
I20260925 05:11:57.315896 28113 raft_consensus.cc:2454] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 263d9c7e012d41bd91ce9e0f5f91d0f0 in term 0.
I20260925 05:11:57.316207 28178 leader_election.cc:302] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263d9c7e012d41bd91ce9e0f5f91d0f0, 79e2ec9af1c448dda97fc79b9878261a; no voters:
I20260925 05:11:57.316632 28312 raft_consensus.cc:2781] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:57.316812 28312 raft_consensus.cc:482] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:57.316927 28312 raft_consensus.cc:3037] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:57.319222 28312 raft_consensus.cc:504] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.319917 28312 leader_election.cc:288] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [CANDIDATE]: Term 1 election: Requested vote from peers 79e2ec9af1c448dda97fc79b9878261a (127.8.215.1:38879), a430ef1e1448423ba4179d4ba6139530 (127.8.215.2:36773)
I20260925 05:11:57.320322 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a327acdf91c4bc08cfa569adec57a26" candidate_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e2ec9af1c448dda97fc79b9878261a"
I20260925 05:11:57.320436 28113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a327acdf91c4bc08cfa569adec57a26" candidate_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a430ef1e1448423ba4179d4ba6139530"
I20260925 05:11:57.320570 27982 raft_consensus.cc:3037] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:57.320658 28113 raft_consensus.cc:3037] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:57.324265 27982 raft_consensus.cc:2454] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263d9c7e012d41bd91ce9e0f5f91d0f0 in term 1.
I20260925 05:11:57.324275 28113 raft_consensus.cc:2454] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263d9c7e012d41bd91ce9e0f5f91d0f0 in term 1.
I20260925 05:11:57.324708 28177 leader_election.cc:302] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263d9c7e012d41bd91ce9e0f5f91d0f0, a430ef1e1448423ba4179d4ba6139530; no voters:
I20260925 05:11:57.325073 28312 raft_consensus.cc:2781] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:57.325522 28312 raft_consensus.cc:686] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 1 LEADER]: Becoming Leader. State: Replica: 263d9c7e012d41bd91ce9e0f5f91d0f0, State: Running, Role: LEADER
I20260925 05:11:57.325994 28312 consensus_queue.cc:237] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } }
I20260925 05:11:57.330513 27841 catalog_manager.cc:5495] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 263d9c7e012d41bd91ce9e0f5f91d0f0 (127.8.215.3). New cstate: current_term: 1 leader_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263d9c7e012d41bd91ce9e0f5f91d0f0" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20260925 05:11:57.405074 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.399610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.8.215.3:33651)
0925 05:11:57.399899 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.399904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.399922 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:11:57.400023 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.400026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.400108 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.400225 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:57.400546 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.401288 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.402815 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.403218 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.403306 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.403874 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:57.404377 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:57.404503 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.404712 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.404777 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260925 05:11:57.412906 28113 raft_consensus.cc:1261] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 1 FOLLOWER]: Refusing update from remote peer 263d9c7e012d41bd91ce9e0f5f91d0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:57.412904 27982 raft_consensus.cc:1261] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a [term 1 FOLLOWER]: Refusing update from remote peer 263d9c7e012d41bd91ce9e0f5f91d0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:57.413877 28312 consensus_queue.cc:1035] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a430ef1e1448423ba4179d4ba6139530" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:57.414167 28316 consensus_queue.cc:1035] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e2ec9af1c448dda97fc79b9878261a" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:57.439527 27840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:57.439801 27840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:57.443337 27840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2eb17b489db4781b1a0767d16ee89f0: Sending DeleteTablet for 3 replicas of tablet 5a327acdf91c4bc08cfa569adec57a26
I20260925 05:11:57.444463 28093 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a327acdf91c4bc08cfa569adec57a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:57 UTC) from {username='slave'} at 127.0.0.1:34144
I20260925 05:11:57.444468 27962 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a327acdf91c4bc08cfa569adec57a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:57 UTC) from {username='slave'} at 127.0.0.1:51298
I20260925 05:11:57.445183 28223 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a327acdf91c4bc08cfa569adec57a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:57 UTC) from {username='slave'} at 127.0.0.1:44896
I20260925 05:11:57.445408 28329 tablet_replica.cc:330] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: stopping tablet replica
I20260925 05:11:57.445930 28329 raft_consensus.cc:2229] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:57.446609 28329 raft_consensus.cc:2258] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:57.448911 28329 ts_tablet_manager.cc:1905] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:57.449309 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27900
I20260925 05:11:57.457237 28329 ts_tablet_manager.cc:1918] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:57.457554 28329 log.cc:1196] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/5a327acdf91c4bc08cfa569adec57a26
I20260925 05:11:57.458299 28329 ts_tablet_manager.cc:1939] T 5a327acdf91c4bc08cfa569adec57a26 P a430ef1e1448423ba4179d4ba6139530: Deleting consensus metadata
I20260925 05:11:57.459636 28331 tablet_replica.cc:330] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: stopping tablet replica
I20260925 05:11:57.460485 27826 catalog_manager.cc:4841] TS a430ef1e1448423ba4179d4ba6139530 (127.8.215.2:36773): tablet 5a327acdf91c4bc08cfa569adec57a26 (table test-table [id=3f7f896fdc0e4c2295ffbba7f01893d5]) successfully deleted
I20260925 05:11:57.460094 28331 raft_consensus.cc:2229] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:57.461525 28331 raft_consensus.cc:2258] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:57.464895 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.450177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.8.215.1:38879)
0925 05:11:57.450369 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.450378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.455242 (+ 4864us) server_negotiation.cc:407] Connection header received
0925 05:11:57.456050 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.456056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.456176 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.456254 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:57.457422 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.458561 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.460475 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.461687 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.461819 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.463289 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:57.463300 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:57.463317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:57.464008 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:57.464061 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:57.464085 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:57.464159 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.464392 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.464501 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:11:57.466164 28022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:57.473979 28331 ts_tablet_manager.cc:1905] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:57.481837 28331 ts_tablet_manager.cc:1918] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:57.482031 28331 log.cc:1196] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/5a327acdf91c4bc08cfa569adec57a26
I20260925 05:11:57.482583 28331 ts_tablet_manager.cc:1939] T 5a327acdf91c4bc08cfa569adec57a26 P 263d9c7e012d41bd91ce9e0f5f91d0f0: Deleting consensus metadata
I20260925 05:11:57.484577 27826 catalog_manager.cc:4841] TS 263d9c7e012d41bd91ce9e0f5f91d0f0 (127.8.215.3:33651): tablet 5a327acdf91c4bc08cfa569adec57a26 (table test-table [id=3f7f896fdc0e4c2295ffbba7f01893d5]) successfully deleted
I20260925 05:11:57.570713 28330 tablet_replica.cc:330] T 5a327acdf91c4bc08cfa569adec57a26 P 79e2ec9af1c448dda97fc79b9878261a: stopping tablet replica
I20260925 05:11:57.570706 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27900
I20260925 05:11:57.594671 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28031
W20260925 05:11:57.594735 27827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.215.1:38879 (error 108)
I20260925 05:11:57.597517 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.596843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51316)
0925 05:11:57.597264 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.597371 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":221,"threads_started":1}
W20260925 05:11:57.597832 27827 catalog_manager.cc:4584] TS 79e2ec9af1c448dda97fc79b9878261a (127.8.215.1:38879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a327acdf91c4bc08cfa569adec57a26: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
I20260925 05:11:57.602849 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.595541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.8.215.2:36773)
0925 05:11:57.595736 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.595744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.595765 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:11:57.595994 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.596000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.596118 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.596229 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:57.596647 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.597820 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.599753 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.600148 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.600242 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.601459 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:57.601467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:57.601480 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:57.602061 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:57.602105 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:57.602124 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:57.602215 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.602387 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.602468 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:11:57.603601 28153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:57.651783 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.651212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51324)
0925 05:11:57.651480 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.651595 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 05:11:57.652988 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.652525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51340)
0925 05:11:57.652708 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.652828 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:11:57.693470 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28031
I20260925 05:11:57.713400 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.712973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51346)
0925 05:11:57.713168 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.713252 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:11:57.714489 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.714126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51354)
0925 05:11:57.714289 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.714356 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:11:57.717239 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28161
I20260925 05:11:57.724969 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.717981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.8.215.3:33651)
0925 05:11:57.718136 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:57.718141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:57.718224 (+ 83us) server_negotiation.cc:407] Connection header received
0925 05:11:57.718390 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:57.718393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:57.718468 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:57.718548 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:57.719013 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.719785 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.721802 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:57.722256 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:57.722352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:57.723503 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:57.723512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:57.723526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:57.724094 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:57.724137 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:57.724158 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:57.724244 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:57.724409 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 05:11:57.724484 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:11:57.725658 28283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:57.795164 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.794692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51364)
0925 05:11:57.794924 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.795026 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 05:11:57.796237 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.795829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51376)
0925 05:11:57.796038 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.796103 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 05:11:57.845688 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28161
I20260925 05:11:57.864884 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 27810
I20260925 05:11:57.865967 27871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:11:57.977303 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:57.976699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:38879 (local address 127.0.0.1:51390)
0925 05:11:57.976944 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:57.977107 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 05:11:57.977363 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 27810
2026-09-25T05:11:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3025 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:11:58 dist-test-slave-wpdm krb5kdc[28341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:11:58 dist-test-slave-wpdm krb5kdc[28341](info): set up 2 sockets
Sep 25 05:11:58 dist-test-slave-wpdm krb5kdc[28341](info): commencing operation
krb5kdc: starting...
W20260925 05:11:59.101923 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:11:59 dist-test-slave-wpdm krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:11:59.372931 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44687
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:44687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:59.705339 28354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:59.705657 28354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:59.705747 28354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:59.713006 28354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:59.713115 28354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:59.713183 28354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:59.713249 28354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:59.713287 28354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:59.723186 28354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:44687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:11:59.724454 28354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:59.725710 28354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:59.737460 28359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:59.737378 28360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:59.737411 28362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:59.738587 28354 server_base.cc:1034] running on GCE node
I20260925 05:11:59.739368 28354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:59.740824 28354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:59.742043 28354 hybrid_clock.cc:648] HybridClock initialized: now 1790313119741985 us; error 53 us; skew 500 ppm
I20260925 05:11:59.742506 28354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:11:59 dist-test-slave-wpdm krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313119, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:59.745520 28354 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:11:59.746968 28354 webserver.cc:460] Webserver started at http://127.8.215.62:41433/ using document root <none> and password file <none>
I20260925 05:11:59.747597 28354 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:59.747705 28354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:59.747957 28354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:59.750851 28354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "51e86b26595e4230821055e2c57206fa"
format_stamp: "Formatted at 2026-09-25 05:11:59 on dist-test-slave-wpdm"
server_key: "09d93f177d76864eecabe67476a68d6f"
server_key_iv: "a8e198ab5854e15cffb47480699d45b7"
server_key_version: "encryptionkey@0"
I20260925 05:11:59.751538 28354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "51e86b26595e4230821055e2c57206fa"
format_stamp: "Formatted at 2026-09-25 05:11:59 on dist-test-slave-wpdm"
server_key: "09d93f177d76864eecabe67476a68d6f"
server_key_iv: "a8e198ab5854e15cffb47480699d45b7"
server_key_version: "encryptionkey@0"
I20260925 05:11:59.756879 28354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:11:59.760245 28369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:59.761873 28354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:59.762068 28354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "51e86b26595e4230821055e2c57206fa"
format_stamp: "Formatted at 2026-09-25 05:11:59 on dist-test-slave-wpdm"
server_key: "09d93f177d76864eecabe67476a68d6f"
server_key_iv: "a8e198ab5854e15cffb47480699d45b7"
server_key_version: "encryptionkey@0"
I20260925 05:11:59.762265 28354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:59.776088 28354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:59.780158 28354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:59.780457 28354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:59.798097 28354 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:44687
I20260925 05:11:59.798121 28420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:44687 every 8 connection(s)
I20260925 05:11:59.799592 28354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:59.803318 28421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:59.809664 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28354
I20260925 05:11:59.809855 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:11:59.810297 9052 external_mini_cluster.cc:1428] Setting key 23f3153d575cac64c681cc5e5c8ca745
I20260925 05:11:59.814005 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa: Bootstrap starting.
I20260925 05:11:59.817766 28421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:59.818864 28421 log.cc:824] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:59.822170 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa: No bootstrap required, opened a new log
Sep 25 05:11:59 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:11:59.830765 28421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e86b26595e4230821055e2c57206fa" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44687 } }
I20260925 05:11:59.831171 28421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:59.831319 28421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e86b26595e4230821055e2c57206fa, State: Initialized, Role: FOLLOWER
I20260925 05:11:59.831979 28421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e86b26595e4230821055e2c57206fa" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44687 } }
I20260925 05:11:59.832242 28421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:59.832401 28421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:59.832602 28421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:59.834604 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:59.812259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.8.215.62:44687)
0925 05:11:59.813080 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:59.813092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:59.813132 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:11:59.814980 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:59.815005 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:59.815435 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:59.815779 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:59.817598 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:59.818985 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:59.820717 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:59.821482 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:59.821615 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:59.826539 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:59.826547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:59.826562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:59.826597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:59.829867 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:59.830567 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:59.830574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:59.830580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:59.830741 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:59.831129 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:59.831137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:59.831140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:59.831840 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:59.832152 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:59.832607 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 05:11:59.833113 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":353,"threads_started":1}
I20260925 05:11:59.835719 28421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e86b26595e4230821055e2c57206fa" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44687 } }
I20260925 05:11:59.836290 28421 leader_election.cc:302] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e86b26595e4230821055e2c57206fa; no voters:
I20260925 05:11:59.837004 28421 leader_election.cc:288] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:59.837108 28426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:59.838336 28426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [term 1 LEADER]: Becoming Leader. State: Replica: 51e86b26595e4230821055e2c57206fa, State: Running, Role: LEADER
I20260925 05:11:59.838841 28426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e86b26595e4230821055e2c57206fa" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44687 } }
I20260925 05:11:59.839037 28421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:59.841410 28428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e86b26595e4230821055e2c57206fa. Latest consensus state: current_term: 1 leader_uuid: "51e86b26595e4230821055e2c57206fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e86b26595e4230821055e2c57206fa" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44687 } } }
I20260925 05:11:59.841835 28428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:59.842007 28427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e86b26595e4230821055e2c57206fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e86b26595e4230821055e2c57206fa" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 44687 } } }
I20260925 05:11:59.842299 28427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:59.844424 28432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:59.849870 28432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:59.859002 28432 catalog_manager.cc:1347] Generated new cluster ID: 4701180071524f9095d4a4e40da1754d
I20260925 05:11:59.859127 28432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:59.899464 28432 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:59.900954 28432 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:59.916832 28432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa: Generated new TSK 0
I20260925 05:11:59.917754 28432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.215.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
Entry for principal kudu/127.8.215.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab.
I20260925 05:11:59.992642 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44687
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:00.359651 28447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:00.359925 28447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:00.360018 28447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:00.367594 28447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:00.367700 28447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:00.367815 28447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:12:00.377767 28447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.1.keytab
--principal=kudu/127.8.215.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:00.378988 28447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:00.380225 28447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:00.393461 28453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:00.393520 28455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:00.393553 28452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:00.394449 28447 server_base.cc:1034] running on GCE node
I20260925 05:12:00.395140 28447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:00.396116 28447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:00.397354 28447 hybrid_clock.cc:648] HybridClock initialized: now 1790313120397318 us; error 58 us; skew 500 ppm
I20260925 05:12:00.397681 28447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:00 dist-test-slave-wpdm krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313120, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:00.400975 28447 init.cc:377] Logged in from keytab as kudu/127.8.215.1@KRBTEST.COM (short username kudu)
I20260925 05:12:00.402346 28447 webserver.cc:460] Webserver started at http://127.8.215.1:32987/ using document root <none> and password file <none>
I20260925 05:12:00.403005 28447 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:00.403115 28447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:00.403362 28447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:00.406253 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "99ca0284b2d14fcb86a89ba60a9fc19f"
format_stamp: "Formatted at 2026-09-25 05:12:00 on dist-test-slave-wpdm"
server_key: "b6a375c9bb122df2d82be600e16bc292"
server_key_iv: "a88d5e678846cb1fd1e0eaa726a471e4"
server_key_version: "encryptionkey@0"
I20260925 05:12:00.406984 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "99ca0284b2d14fcb86a89ba60a9fc19f"
format_stamp: "Formatted at 2026-09-25 05:12:00 on dist-test-slave-wpdm"
server_key: "b6a375c9bb122df2d82be600e16bc292"
server_key_iv: "a88d5e678846cb1fd1e0eaa726a471e4"
server_key_version: "encryptionkey@0"
I20260925 05:12:00.411901 28447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:12:00.415218 28462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:00.416570 28447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:12:00.416703 28447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "99ca0284b2d14fcb86a89ba60a9fc19f"
format_stamp: "Formatted at 2026-09-25 05:12:00 on dist-test-slave-wpdm"
server_key: "b6a375c9bb122df2d82be600e16bc292"
server_key_iv: "a88d5e678846cb1fd1e0eaa726a471e4"
server_key_version: "encryptionkey@0"
I20260925 05:12:00.416992 28447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:00.432258 28447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:00.436618 28447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:00.436959 28447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:00.437913 28447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:00.439324 28447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:00.439395 28447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:00.439486 28447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:00.439532 28447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:00.473242 28447 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:33543
I20260925 05:12:00.473330 28574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:33543 every 8 connection(s)
I20260925 05:12:00.474622 28447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:00.484181 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28447
I20260925 05:12:00.484347 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:12:00.484858 9052 external_mini_cluster.cc:1428] Setting key 9c895fe3913807d8f201cc2acb41e8b8
Sep 25 05:12:00 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313120, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.1@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:00.496093 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:00.477169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:49497 (local address 127.8.215.62:44687)
0925 05:12:00.477671 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:00.477677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:00.478668 (+ 991us) server_negotiation.cc:407] Connection header received
0925 05:12:00.480518 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:00.480523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:00.480629 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:00.480768 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:00.482345 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:00.483066 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:00.484742 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:00.485216 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:00.485357 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:00.489407 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:00.489412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:00.489415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:00.489446 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:00.492852 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:00.493654 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:00.493661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:00.493664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:00.493820 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:00.494301 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:00.494307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:00.494310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:00.494584 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:00.494751 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:00.495446 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 05:12:00.495678 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260925 05:12:00.497265 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:00.477754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44687 (local address 127.8.215.1:49497)
0925 05:12:00.478501 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:00.478539 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:00.480236 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:00.481016 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:00.481025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:00.481571 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:00.482146 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:00.482165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:00.483227 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:00.483232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:00.484585 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:00.484597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:00.485452 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:00.485461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:00.485715 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:00.486478 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:00.486511 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:00.489174 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:00.493010 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:00.493020 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:00.493052 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:00.493518 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:00.493941 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:00.493946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:00.493950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:00.494182 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:00.494712 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:00.494721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:00.495075 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 05:12:00.495352 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 05:12:00.495920 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":325,"threads_started":1}
I20260925 05:12:00.498971 28575 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44687
I20260925 05:12:00.499359 28575 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:00.500279 28575 heartbeater.cc:511] Master 127.8.215.62:44687 requested a full tablet report, sending...
I20260925 05:12:00.502990 28386 ts_manager.cc:194] Registered new tserver with Master: 99ca0284b2d14fcb86a89ba60a9fc19f (127.8.215.1:33543)
I20260925 05:12:00.504594 28386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.1@KRBTEST.COM'} at 127.8.215.1:49497
WARNING: no policy specified for kudu/127.8.215.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
Entry for principal kudu/127.8.215.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab.
I20260925 05:12:00.552098 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44687
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:00.880512 28581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:00.880858 28581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:00.881067 28581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:00.888370 28581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:00.888471 28581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:00.888665 28581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:12:00.898320 28581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.2.keytab
--principal=kudu/127.8.215.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:00.899570 28581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:00.900906 28581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:00.913949 28586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:00.914371 28587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:00.915117 28581 server_base.cc:1034] running on GCE node
W20260925 05:12:00.915252 28589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:00.915951 28581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:00.916976 28581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:00.918205 28581 hybrid_clock.cc:648] HybridClock initialized: now 1790313120918147 us; error 53 us; skew 500 ppm
I20260925 05:12:00.918520 28581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:00 dist-test-slave-wpdm krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313120, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:00.921803 28581 init.cc:377] Logged in from keytab as kudu/127.8.215.2@KRBTEST.COM (short username kudu)
I20260925 05:12:00.923329 28581 webserver.cc:460] Webserver started at http://127.8.215.2:32879/ using document root <none> and password file <none>
I20260925 05:12:00.923969 28581 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:00.924064 28581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:00.924324 28581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:00.927186 28581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "ae913d1818ea4ad6849ea4db9932be6e"
format_stamp: "Formatted at 2026-09-25 05:12:00 on dist-test-slave-wpdm"
server_key: "c990ccdc8b5fd6e4735cbcfa26e795b7"
server_key_iv: "525ca864c024ad8f87fa7f8946006326"
server_key_version: "encryptionkey@0"
I20260925 05:12:00.927909 28581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "ae913d1818ea4ad6849ea4db9932be6e"
format_stamp: "Formatted at 2026-09-25 05:12:00 on dist-test-slave-wpdm"
server_key: "c990ccdc8b5fd6e4735cbcfa26e795b7"
server_key_iv: "525ca864c024ad8f87fa7f8946006326"
server_key_version: "encryptionkey@0"
I20260925 05:12:00.933032 28581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:12:00.937278 28596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:00.939081 28581 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:12:00.939265 28581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "ae913d1818ea4ad6849ea4db9932be6e"
format_stamp: "Formatted at 2026-09-25 05:12:00 on dist-test-slave-wpdm"
server_key: "c990ccdc8b5fd6e4735cbcfa26e795b7"
server_key_iv: "525ca864c024ad8f87fa7f8946006326"
server_key_version: "encryptionkey@0"
I20260925 05:12:00.939527 28581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:00.965204 28581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:00.969307 28581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:00.969615 28581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:00.970606 28581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:00.971922 28581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:00.972028 28581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:00.972174 28581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:00.972256 28581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:01.005488 28581 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:36179
I20260925 05:12:01.005533 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:36179 every 8 connection(s)
I20260925 05:12:01.006837 28581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:12:01.008068 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28581
I20260925 05:12:01.008249 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:12:01.008838 9052 external_mini_cluster.cc:1428] Setting key e3bae6f6a175fcce597696d00ccdbf9d
Sep 25 05:12:01 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313120, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.2@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:01.032474 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.009687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:55269 (local address 127.8.215.62:44687)
0925 05:12:01.010157 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.010161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.011417 (+ 1256us) server_negotiation.cc:407] Connection header received
0925 05:12:01.014413 (+ 2996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.014418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.014485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.014589 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:01.016952 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.017661 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.020308 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.021060 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.021185 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.025259 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:01.025268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:01.025276 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:01.025323 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:01.029013 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.029803 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.029811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.029817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.029977 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.030401 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.030409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.030415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.030845 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:01.030997 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.031715 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.031983 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260925 05:12:01.033099 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.010311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44687 (local address 127.8.215.2:55269)
0925 05:12:01.011225 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.011285 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.014070 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.014960 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.014974 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.015858 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:01.016764 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.016795 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.017825 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.017833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.020142 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.020160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.021310 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.021319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.021692 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.022509 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:01.022543 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:01.024996 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:01.029165 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:01.029172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:01.029186 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:01.029651 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:01.030083 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:01.030086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:01.030089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:01.030276 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:01.030981 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:01.030987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:01.031314 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.031538 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.031935 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":361,"threads_started":1}
I20260925 05:12:01.034758 28709 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44687
I20260925 05:12:01.035109 28709 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:01.036006 28709 heartbeater.cc:511] Master 127.8.215.62:44687 requested a full tablet report, sending...
I20260925 05:12:01.037711 28386 ts_manager.cc:194] Registered new tserver with Master: ae913d1818ea4ad6849ea4db9932be6e (127.8.215.2:36179)
I20260925 05:12:01.038537 28386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.2@KRBTEST.COM'} at 127.8.215.2:55269
WARNING: no policy specified for kudu/127.8.215.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
Entry for principal kudu/127.8.215.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab.
I20260925 05:12:01.075920 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44687
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:01.411257 28715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:01.411538 28715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:01.411710 28715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:01.419534 28715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:01.419687 28715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:01.419873 28715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:12:01.429946 28715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.3.keytab
--principal=kudu/127.8.215.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:01.431273 28715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:01.432572 28715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:01.446262 28720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:01.446182 28723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:01.446229 28721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:01.447327 28715 server_base.cc:1034] running on GCE node
I20260925 05:12:01.447899 28715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:01.448930 28715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:01.450129 28715 hybrid_clock.cc:648] HybridClock initialized: now 1790313121450072 us; error 45 us; skew 500 ppm
I20260925 05:12:01.450419 28715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:01 dist-test-slave-wpdm krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313121, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:01.453573 28715 init.cc:377] Logged in from keytab as kudu/127.8.215.3@KRBTEST.COM (short username kudu)
I20260925 05:12:01.455081 28715 webserver.cc:460] Webserver started at http://127.8.215.3:33595/ using document root <none> and password file <none>
I20260925 05:12:01.455802 28715 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:01.455950 28715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:01.456265 28715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:01.459736 28715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f"
format_stamp: "Formatted at 2026-09-25 05:12:01 on dist-test-slave-wpdm"
server_key: "a513097f7566aa3f89f3b77ca49d53f3"
server_key_iv: "460b4cacd02fedc678829821de7222b6"
server_key_version: "encryptionkey@0"
I20260925 05:12:01.460551 28715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f"
format_stamp: "Formatted at 2026-09-25 05:12:01 on dist-test-slave-wpdm"
server_key: "a513097f7566aa3f89f3b77ca49d53f3"
server_key_iv: "460b4cacd02fedc678829821de7222b6"
server_key_version: "encryptionkey@0"
I20260925 05:12:01.465909 28715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 05:12:01.469449 28730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:01.470736 28715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:12:01.470914 28715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f"
format_stamp: "Formatted at 2026-09-25 05:12:01 on dist-test-slave-wpdm"
server_key: "a513097f7566aa3f89f3b77ca49d53f3"
server_key_iv: "460b4cacd02fedc678829821de7222b6"
server_key_version: "encryptionkey@0"
I20260925 05:12:01.471123 28715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:01.496269 28715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:01.500797 28715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:01.501101 28715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:01.502112 28715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:01.503427 28715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:01.503530 28715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:01.503676 28715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:01.503747 28715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:01.508561 28575 heartbeater.cc:503] Master 127.8.215.62:44687 was elected leader, sending a full tablet report...
I20260925 05:12:01.538123 28715 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:33357
I20260925 05:12:01.538187 28842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:33357 every 8 connection(s)
I20260925 05:12:01.539712 28715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:01.543131 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28715
I20260925 05:12:01.543354 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:12:01.543941 9052 external_mini_cluster.cc:1428] Setting key 8f3923555f4c8015a3d99d568eb779d9
Sep 25 05:12:01 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313121, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.3@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:01.560988 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.542549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:39423 (local address 127.8.215.62:44687)
0925 05:12:01.543115 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.543124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.544362 (+ 1238us) server_negotiation.cc:407] Connection header received
0925 05:12:01.546529 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.546535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.546618 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.546755 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:01.548337 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.549216 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.550791 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.551216 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.551355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.554892 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:01.554896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:01.554900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:01.554933 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:01.557438 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.558310 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.558317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.558321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.558451 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.559004 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.559012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.559017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.559412 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:01.559530 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.560429 (+ 899us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.560597 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":250,"threads_started":1}
I20260925 05:12:01.561995 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.543186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:44687 (local address 127.8.215.3:39423)
0925 05:12:01.544165 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.544220 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.546265 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.547038 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.547047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.547579 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:01.548167 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.548185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.549368 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.549374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.550619 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.550630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.551476 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.551483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.551724 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.552351 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:01.552373 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:01.554669 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:01.557577 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:01.557588 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:01.557608 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:01.558191 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:01.558562 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:01.558567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:01.558571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:01.558858 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:01.559675 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:01.559687 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:01.560084 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.560334 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.560840 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":391,"threads_started":1}
I20260925 05:12:01.563429 28843 heartbeater.cc:347] Connected to a master server at 127.8.215.62:44687
I20260925 05:12:01.563745 28843 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:01.564477 28843 heartbeater.cc:511] Master 127.8.215.62:44687 requested a full tablet report, sending...
I20260925 05:12:01.566092 28386 ts_manager.cc:194] Registered new tserver with Master: e5b39c5e0c2a4fc6ab6a71189f58ff2f (127.8.215.3:33357)
I20260925 05:12:01.566979 28386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.215.3@KRBTEST.COM'} at 127.8.215.3:39423
I20260925 05:12:01.571780 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:12:01.589077 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.576227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.8.215.62:44687)
0925 05:12:01.576391 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.576395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.576805 (+ 410us) server_negotiation.cc:407] Connection header received
0925 05:12:01.576944 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.576947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.577021 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.577155 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:01.578501 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.579246 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.580544 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.581119 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.581233 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.583686 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:01.583692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:01.583695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:01.583728 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:01.586449 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.587072 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.587077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.587081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.587238 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.587608 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.587612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.587614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.587983 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:01.588229 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.588529 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.588716 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:12:01.592914 28386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:12:01.599823 28386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:12:01.623380 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.612319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36179 (local address 127.0.0.1:54394)
0925 05:12:01.613864 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.613913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.614243 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.615346 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.615353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.615392 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:01.615767 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.615776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.617874 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.617881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.620227 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.620242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.622394 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.622400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.622597 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.622654 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.622793 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.622988 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":1029,"threads_started":1}
I20260925 05:12:01.623384 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.612521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.8.215.2:36179)
0925 05:12:01.613179 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.613193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.614043 (+ 850us) server_negotiation.cc:407] Connection header received
0925 05:12:01.614521 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.614531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.614821 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.615050 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:01.615947 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.617732 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.620417 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.622178 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.622335 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.622582 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.622854 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.623042 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":294,"threads_started":1}
I20260925 05:12:01.624183 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.612608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:33543 (local address 127.0.0.1:57436)
0925 05:12:01.614617 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.614637 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.615925 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.616598 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.616604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.616631 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:01.617037 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.617051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.618955 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.618960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.621508 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.621524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.623309 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.623315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.623515 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.623564 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.623639 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.623761 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1875,"mutex_wait_us":42,"thread_start_us":1695,"threads_started":1}
I20260925 05:12:01.624933 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.612992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.8.215.1:33543)
0925 05:12:01.614132 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.614142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.614736 (+ 594us) server_negotiation.cc:407] Connection header received
0925 05:12:01.616102 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.616112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.616377 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.616576 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:01.617264 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.618804 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.621667 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.622858 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.622970 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.623189 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.624412 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.624574 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":1025,"threads_started":1}
I20260925 05:12:01.628278 28644 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8bcf26943845bfafea1767f1b00a73 (DEFAULT_TABLE table=test-table [id=dfb21ff551e4433eb736b31a2ce8ee90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:01.628702 28510 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8bcf26943845bfafea1767f1b00a73 (DEFAULT_TABLE table=test-table [id=dfb21ff551e4433eb736b31a2ce8ee90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:01.629407 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.614811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:33357 (local address 127.0.0.1:53146)
0925 05:12:01.615377 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.615393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.615502 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.617587 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.617591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.617613 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:01.618122 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.618139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.619819 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.619825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.622356 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.622376 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.624661 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.624668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.624885 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.624925 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.628902 (+ 3977us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.629033 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":76,"thread_start_us":289,"threads_started":1}
I20260925 05:12:01.629432 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.615023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.8.215.3:33357)
0925 05:12:01.616478 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.616489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.616523 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:12:01.616612 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.616619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.617123 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.617325 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:01.618344 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.619663 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.622550 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.624086 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.624224 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.624489 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.628942 (+ 4453us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.629109 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1279,"thread_start_us":1127,"threads_started":1}
I20260925 05:12:01.630520 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8bcf26943845bfafea1767f1b00a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:01.630735 28644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8bcf26943845bfafea1767f1b00a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:01.632958 28778 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8bcf26943845bfafea1767f1b00a73 (DEFAULT_TABLE table=test-table [id=dfb21ff551e4433eb736b31a2ce8ee90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:01.634224 28778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8bcf26943845bfafea1767f1b00a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:01.645087 28862 tablet_bootstrap.cc:492] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Bootstrap starting.
I20260925 05:12:01.647469 28863 tablet_bootstrap.cc:492] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Bootstrap starting.
I20260925 05:12:01.649484 28864 tablet_bootstrap.cc:492] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Bootstrap starting.
I20260925 05:12:01.649832 28862 tablet_bootstrap.cc:654] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:01.650828 28863 tablet_bootstrap.cc:654] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:01.651113 28862 log.cc:824] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:01.652041 28863 log.cc:824] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:01.653143 28864 tablet_bootstrap.cc:654] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:01.654671 28862 tablet_bootstrap.cc:492] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: No bootstrap required, opened a new log
I20260925 05:12:01.654805 28864 log.cc:824] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:01.654991 28862 ts_tablet_manager.cc:1397] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 05:12:01.657990 28864 tablet_bootstrap.cc:492] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: No bootstrap required, opened a new log
I20260925 05:12:01.658314 28864 ts_tablet_manager.cc:1397] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 05:12:01.658442 28863 tablet_bootstrap.cc:492] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: No bootstrap required, opened a new log
I20260925 05:12:01.658722 28863 ts_tablet_manager.cc:1397] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260925 05:12:01.667044 28864 raft_consensus.cc:348] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.667068 28862 raft_consensus.cc:348] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.667497 28864 raft_consensus.cc:374] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:01.667554 28862 raft_consensus.cc:374] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:01.667598 28864 raft_consensus.cc:729] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b39c5e0c2a4fc6ab6a71189f58ff2f, State: Initialized, Role: FOLLOWER
I20260925 05:12:01.667671 28862 raft_consensus.cc:729] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae913d1818ea4ad6849ea4db9932be6e, State: Initialized, Role: FOLLOWER
I20260925 05:12:01.668278 28864 consensus_queue.cc:260] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.668402 28862 consensus_queue.cc:260] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.669842 28709 heartbeater.cc:503] Master 127.8.215.62:44687 was elected leader, sending a full tablet report...
I20260925 05:12:01.670481 28862 ts_tablet_manager.cc:1428] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 05:12:01.671296 28843 heartbeater.cc:503] Master 127.8.215.62:44687 was elected leader, sending a full tablet report...
I20260925 05:12:01.672040 28864 ts_tablet_manager.cc:1428] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260925 05:12:01.672413 28863 raft_consensus.cc:348] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.673507 28863 raft_consensus.cc:374] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:01.673650 28863 raft_consensus.cc:729] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ca0284b2d14fcb86a89ba60a9fc19f, State: Initialized, Role: FOLLOWER
I20260925 05:12:01.674440 28863 consensus_queue.cc:260] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.676409 28863 ts_tablet_manager.cc:1428] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
W20260925 05:12:01.727154 28576 tablet.cc:2369] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:12:01.758895 28710 tablet.cc:2369] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:01.767036 28870 raft_consensus.cc:482] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:01.767400 28870 raft_consensus.cc:504] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.769255 28870 leader_election.cc:288] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae913d1818ea4ad6849ea4db9932be6e (127.8.215.2:36179), e5b39c5e0c2a4fc6ab6a71189f58ff2f (127.8.215.3:33357)
I20260925 05:12:01.776347 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.769665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:39039 (local address 127.8.215.3:33357)
0925 05:12:01.769884 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.769892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.770839 (+ 947us) server_negotiation.cc:407] Connection header received
0925 05:12:01.770982 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.770985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.771042 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.771110 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:01.771707 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.772433 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.774477 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.775304 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.775457 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.775560 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.775930 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.776075 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:12:01.776470 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.769779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:40221 (local address 127.8.215.2:36179)
0925 05:12:01.769991 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.770000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.770631 (+ 631us) server_negotiation.cc:407] Connection header received
0925 05:12:01.770780 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.770789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.770873 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.770958 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:01.771830 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.772518 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.774551 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.775487 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.775604 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.775699 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.776095 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.776223 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:12:01.776510 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.769656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36179 (local address 127.8.215.1:40221)
0925 05:12:01.770516 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.770531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.770647 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.771258 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.771261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.771292 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:01.771674 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.771682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.772656 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.772665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.774346 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.774358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.775780 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.775785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.775943 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.775971 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.776050 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.776168 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":1,"thread_start_us":600,"threads_started":1}
I20260925 05:12:01.776818 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.769704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:33357 (local address 127.8.215.1:39039)
0925 05:12:01.770730 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:01.770748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:01.770868 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:01.771166 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:01.771169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:01.771189 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:01.771518 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.771524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.772561 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.772569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.774298 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:01.774313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.775513 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.775518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:01.775742 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.775782 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:12:01.775872 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:12:01.776003 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":3,"thread_start_us":414,"threads_started":1}
I20260925 05:12:01.777500 28664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8bcf26943845bfafea1767f1b00a73" candidate_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae913d1818ea4ad6849ea4db9932be6e" is_pre_election: true
I20260925 05:12:01.777627 28798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8bcf26943845bfafea1767f1b00a73" candidate_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" is_pre_election: true
I20260925 05:12:01.778029 28664 raft_consensus.cc:2454] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ca0284b2d14fcb86a89ba60a9fc19f in term 0.
I20260925 05:12:01.778162 28798 raft_consensus.cc:2454] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ca0284b2d14fcb86a89ba60a9fc19f in term 0.
I20260925 05:12:01.778764 28464 leader_election.cc:302] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 99ca0284b2d14fcb86a89ba60a9fc19f, ae913d1818ea4ad6849ea4db9932be6e, e5b39c5e0c2a4fc6ab6a71189f58ff2f; no voters:
I20260925 05:12:01.779228 28870 raft_consensus.cc:2781] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:12:01.779372 28870 raft_consensus.cc:482] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:12:01.779474 28870 raft_consensus.cc:3037] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:01.781646 28870 raft_consensus.cc:504] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
I20260925 05:12:01.782272 28870 leader_election.cc:288] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [CANDIDATE]: Term 1 election: Requested vote from peers ae913d1818ea4ad6849ea4db9932be6e (127.8.215.2:36179), e5b39c5e0c2a4fc6ab6a71189f58ff2f (127.8.215.3:33357)
I20260925 05:12:01.782661 28664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8bcf26943845bfafea1767f1b00a73" candidate_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae913d1818ea4ad6849ea4db9932be6e"
I20260925 05:12:01.782862 28664 raft_consensus.cc:3037] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:01.782786 28798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8bcf26943845bfafea1767f1b00a73" candidate_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f"
I20260925 05:12:01.783043 28798 raft_consensus.cc:3037] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:01.785084 28664 raft_consensus.cc:2454] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ca0284b2d14fcb86a89ba60a9fc19f in term 1.
I20260925 05:12:01.785339 28798 raft_consensus.cc:2454] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ca0284b2d14fcb86a89ba60a9fc19f in term 1.
I20260925 05:12:01.785526 28464 leader_election.cc:302] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ca0284b2d14fcb86a89ba60a9fc19f, ae913d1818ea4ad6849ea4db9932be6e; no voters:
I20260925 05:12:01.785861 28870 raft_consensus.cc:2781] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:01.786315 28870 raft_consensus.cc:686] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 1 LEADER]: Becoming Leader. State: Replica: 99ca0284b2d14fcb86a89ba60a9fc19f, State: Running, Role: LEADER
I20260925 05:12:01.786799 28870 consensus_queue.cc:237] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } }
W20260925 05:12:01.791389 28844 tablet.cc:2369] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:01.791708 28385 catalog_manager.cc:5495] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ca0284b2d14fcb86a89ba60a9fc19f (127.8.215.1). New cstate: current_term: 1 leader_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ca0284b2d14fcb86a89ba60a9fc19f" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 33543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 } health_report { overall_health: UNKNOWN } } }
I20260925 05:12:01.816368 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.811285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.8.215.1:33543)
0925 05:12:01.811485 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.811491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.811557 (+ 66us) server_negotiation.cc:407] Connection header received
0925 05:12:01.811740 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.811745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.811864 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.812022 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:01.812372 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.813177 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.814499 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.814880 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.814969 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.815385 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:01.815914 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:01.816020 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.816108 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.816175 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:12:01.824419 28664 raft_consensus.cc:1261] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 1 FOLLOWER]: Refusing update from remote peer 99ca0284b2d14fcb86a89ba60a9fc19f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:01.824419 28798 raft_consensus.cc:1261] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 1 FOLLOWER]: Refusing update from remote peer 99ca0284b2d14fcb86a89ba60a9fc19f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:01.825457 28873 consensus_queue.cc:1035] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b39c5e0c2a4fc6ab6a71189f58ff2f" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:01.825783 28870 consensus_queue.cc:1035] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae913d1818ea4ad6849ea4db9932be6e" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:01.838950 28877 mvcc.cc:205] Tried to move back new op lower bound from 7333122546983186432 to 7333122546842337280. Current Snapshot: MvccSnapshot[applied={T|T < 7333122546983186432}]
I20260925 05:12:01.852632 28386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:01.852988 28386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:01.856595 28386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e86b26595e4230821055e2c57206fa: Sending DeleteTablet for 3 replicas of tablet 5a8bcf26943845bfafea1767f1b00a73
I20260925 05:12:01.857851 28644 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8bcf26943845bfafea1767f1b00a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:54394
I20260925 05:12:01.857986 28778 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8bcf26943845bfafea1767f1b00a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:53146
I20260925 05:12:01.858188 28510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8bcf26943845bfafea1767f1b00a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:01 UTC) from {username='kudu', principal='kudu/127.8.215.62@KRBTEST.COM'} at 127.0.0.1:57436
I20260925 05:12:01.859845 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28447
I20260925 05:12:01.870631 28886 tablet_replica.cc:330] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: stopping tablet replica
I20260925 05:12:01.871335 28886 raft_consensus.cc:2229] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:01.871896 28888 tablet_replica.cc:330] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: stopping tablet replica
I20260925 05:12:01.872081 28886 raft_consensus.cc:2258] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:01.872416 28888 raft_consensus.cc:2229] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:12:01.873162 28888 raft_consensus.cc:2258] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:01.874219 28887 tablet_replica.cc:330] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: stopping tablet replica
I20260925 05:12:01.874675 28886 ts_tablet_manager.cc:1905] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:01.874708 28887 raft_consensus.cc:2229] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:01.875480 28887 raft_consensus.cc:2258] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:01.877842 28887 ts_tablet_manager.cc:1905] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:01.883457 28886 ts_tablet_manager.cc:1918] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:01.884323 28888 ts_tablet_manager.cc:1905] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:01.885018 28887 ts_tablet_manager.cc:1918] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:01.885185 28887 log.cc:1196] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/5a8bcf26943845bfafea1767f1b00a73
I20260925 05:12:01.885716 28887 ts_tablet_manager.cc:1939] T 5a8bcf26943845bfafea1767f1b00a73 P e5b39c5e0c2a4fc6ab6a71189f58ff2f: Deleting consensus metadata
I20260925 05:12:01.886842 28886 log.cc:1196] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/5a8bcf26943845bfafea1767f1b00a73
I20260925 05:12:01.887446 28886 ts_tablet_manager.cc:1939] T 5a8bcf26943845bfafea1767f1b00a73 P ae913d1818ea4ad6849ea4db9932be6e: Deleting consensus metadata
I20260925 05:12:01.888142 28370 catalog_manager.cc:4841] TS e5b39c5e0c2a4fc6ab6a71189f58ff2f (127.8.215.3:33357): tablet 5a8bcf26943845bfafea1767f1b00a73 (table test-table [id=dfb21ff551e4433eb736b31a2ce8ee90]) successfully deleted
I20260925 05:12:01.889297 28371 catalog_manager.cc:4841] TS ae913d1818ea4ad6849ea4db9932be6e (127.8.215.2:36179): tablet 5a8bcf26943845bfafea1767f1b00a73 (table test-table [id=dfb21ff551e4433eb736b31a2ce8ee90]) successfully deleted
Sep 25 05:12:01 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.1@KRBTEST.COM
I20260925 05:12:01.892830 28888 ts_tablet_manager.cc:1918] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:01.893024 28888 log.cc:1196] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/5a8bcf26943845bfafea1767f1b00a73
I20260925 05:12:01.893571 28888 ts_tablet_manager.cc:1939] T 5a8bcf26943845bfafea1767f1b00a73 P 99ca0284b2d14fcb86a89ba60a9fc19f: Deleting consensus metadata
I20260925 05:12:01.895526 28370 catalog_manager.cc:4841] TS 99ca0284b2d14fcb86a89ba60a9fc19f (127.8.215.1:33543): tablet 5a8bcf26943845bfafea1767f1b00a73 (table test-table [id=dfb21ff551e4433eb736b31a2ce8ee90]) successfully deleted
I20260925 05:12:01.899017 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:01.861519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.8.215.1:33543)
0925 05:12:01.861688 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:01.861696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:01.879267 (+ 17571us) server_negotiation.cc:407] Connection header received
0925 05:12:01.879738 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:01.879744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:01.879824 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:01.879893 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:01.881748 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.882992 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.884980 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:01.885771 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:01.885896 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:01.892339 (+ 6443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:01.892353 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:01.892371 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:01.892416 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:01.896053 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.896808 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.896817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.896821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.896967 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:01.897405 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:01.897409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:01.897413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:01.897796 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:01.898099 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:01.898421 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 05:12:01.898678 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:12:01.899696 28570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:02.002753 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28447
I20260925 05:12:02.028108 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28581
Sep 25 05:12:02 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.2@KRBTEST.COM
I20260925 05:12:02.044263 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:02.028947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.8.215.2:36179)
0925 05:12:02.029198 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:02.029204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:02.029224 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:12:02.029361 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:02.029367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:02.029461 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:02.029565 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:02.030826 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:02.031604 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:02.032921 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:02.033341 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:02.033475 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:02.038066 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:02.038087 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:02.038102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:02.038138 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:02.041283 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:02.042039 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:02.042047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:02.042049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:02.042245 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:02.042635 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:02.042641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:02.042644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:02.043042 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:02.043348 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:02.043659 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 05:12:02.043893 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260925 05:12:02.045086 28704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:02.134712 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28581
I20260925 05:12:02.153070 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28715
Sep 25 05:12:02 dist-test-slave-wpdm krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.3@KRBTEST.COM
I20260925 05:12:02.168442 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:02.153883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.8.215.3:33357)
0925 05:12:02.154052 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:02.154057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:02.154076 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:12:02.154204 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:02.154208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:02.154272 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:02.154359 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:02.155795 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:02.156532 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:02.157875 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:02.158261 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:02.158356 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:02.162831 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:02.162848 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:02.162864 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:02.162907 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:02.165955 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:02.166607 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:02.166620 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:02.166622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:02.166768 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:02.167156 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:02.167160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:02.167162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:02.167532 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:02.167761 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:02.167993 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 05:12:02.168179 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:12:02.169143 28838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:02.258641 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28715
I20260925 05:12:02.276299 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28354
I20260925 05:12:02.277444 28416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:02.389353 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28354
2026-09-25T05:12:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4413 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T05:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:02Z Disabled control of system clock
I20260925 05:12:02.472545 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:38979
--webserver_interface=127.8.215.62
--webserver_port=0
--builtin_ntp_servers=127.8.215.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:38979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:12:02.821673 28897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:02.822042 28897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:02.822212 28897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:02.829918 28897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:02.830080 28897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:02.830163 28897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:02.830289 28897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:02.830375 28897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:02.840348 28897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:38979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:02.841775 28897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:02.843762 28897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:02.856575 28902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:02.856628 28903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:02.856663 28905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:02.857874 28897 server_base.cc:1034] running on GCE node
I20260925 05:12:02.858719 28897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:02.860357 28897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:02.861603 28897 hybrid_clock.cc:648] HybridClock initialized: now 1790313122861553 us; error 49 us; skew 500 ppm
I20260925 05:12:02.861999 28897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:12:02.864487 28897 webserver.cc:460] Webserver started at http://127.8.215.62:41541/ using document root <none> and password file <none>
I20260925 05:12:02.865290 28897 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:02.865427 28897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:02.865754 28897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:02.868861 28897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "89881b4addad47a3918abcd8b69b001d"
format_stamp: "Formatted at 2026-09-25 05:12:02 on dist-test-slave-wpdm"
server_key: "33761cd926ed5d1ae8de8323faf441ba"
server_key_iv: "805b112786d156803c76e35723f19d5c"
server_key_version: "encryptionkey@0"
I20260925 05:12:02.869707 28897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "89881b4addad47a3918abcd8b69b001d"
format_stamp: "Formatted at 2026-09-25 05:12:02 on dist-test-slave-wpdm"
server_key: "33761cd926ed5d1ae8de8323faf441ba"
server_key_iv: "805b112786d156803c76e35723f19d5c"
server_key_version: "encryptionkey@0"
I20260925 05:12:02.875025 28897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 05:12:02.878669 28911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:02.880045 28897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:12:02.880252 28897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "89881b4addad47a3918abcd8b69b001d"
format_stamp: "Formatted at 2026-09-25 05:12:02 on dist-test-slave-wpdm"
server_key: "33761cd926ed5d1ae8de8323faf441ba"
server_key_iv: "805b112786d156803c76e35723f19d5c"
server_key_version: "encryptionkey@0"
I20260925 05:12:02.880481 28897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:02.906423 28897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:02.907311 28897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:02.907615 28897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:02.928004 28897 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:38979
I20260925 05:12:02.928095 28962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:38979 every 8 connection(s)
I20260925 05:12:02.929486 28897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:02.930696 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28897
I20260925 05:12:02.930872 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:02.931335 9052 external_mini_cluster.cc:1428] Setting key 195c36f30cc77730c2f4a909d0de6b90
I20260925 05:12:02.934150 28963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:02.948618 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:02.933266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.8.215.62:38979)
0925 05:12:02.934370 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:02.934382 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:02.934433 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:12:02.937251 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:02.937294 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:02.938015 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:02.938541 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:02.939385 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:02.941670 (+ 2285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:02.943827 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:02.944933 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:02.945129 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:02.946337 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:02.946345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:02.946362 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:02.946935 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:02.947011 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:02.947034 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:02.947161 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:02.947390 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 05:12:02.947697 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":430,"threads_started":1}
I20260925 05:12:02.949249 28963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d: Bootstrap starting.
I20260925 05:12:02.952569 28963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:02.953542 28963 log.cc:824] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:02.956413 28963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d: No bootstrap required, opened a new log
I20260925 05:12:02.964094 28963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89881b4addad47a3918abcd8b69b001d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38979 } }
I20260925 05:12:02.964445 28963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:02.964519 28963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89881b4addad47a3918abcd8b69b001d, State: Initialized, Role: FOLLOWER
I20260925 05:12:02.965086 28963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89881b4addad47a3918abcd8b69b001d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38979 } }
I20260925 05:12:02.965282 28963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:02.965378 28963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:02.965505 28963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:02.967573 28963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89881b4addad47a3918abcd8b69b001d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38979 } }
I20260925 05:12:02.968035 28963 leader_election.cc:302] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89881b4addad47a3918abcd8b69b001d; no voters:
I20260925 05:12:02.968576 28963 leader_election.cc:288] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:02.968763 28968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:02.969527 28968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [term 1 LEADER]: Becoming Leader. State: Replica: 89881b4addad47a3918abcd8b69b001d, State: Running, Role: LEADER
I20260925 05:12:02.970118 28968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89881b4addad47a3918abcd8b69b001d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38979 } }
I20260925 05:12:02.970885 28963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:02.973273 28970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89881b4addad47a3918abcd8b69b001d. Latest consensus state: current_term: 1 leader_uuid: "89881b4addad47a3918abcd8b69b001d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89881b4addad47a3918abcd8b69b001d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38979 } } }
I20260925 05:12:02.973706 28970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:02.973745 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89881b4addad47a3918abcd8b69b001d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89881b4addad47a3918abcd8b69b001d" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38979 } } }
I20260925 05:12:02.974176 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:02.975910 28977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:02.982503 28977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:02.991681 28977 catalog_manager.cc:1347] Generated new cluster ID: 10a3aae1e4ef4f5395fb41f4b20d7809
I20260925 05:12:02.991848 28977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:03.001220 28977 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:03.003022 28977 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:03.021654 28977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d: Generated new TSK 0
I20260925 05:12:03.022785 28977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:03.037837 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.1:0
--local_ip_for_outbound_sockets=127.8.215.1
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38979
--builtin_ntp_servers=127.8.215.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:12:03.382526 28987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:03.382829 28987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:03.382982 28987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:03.390702 28987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:03.390820 28987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:03.391008 28987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.1
I20260925 05:12:03.400590 28987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.215.1
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.1
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:03.402033 28987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:03.403913 28987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:03.421245 28992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:03.421320 28993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:03.421137 28995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:03.422178 28987 server_base.cc:1034] running on GCE node
I20260925 05:12:03.422890 28987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:03.423956 28987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:03.425163 28987 hybrid_clock.cc:648] HybridClock initialized: now 1790313123425092 us; error 62 us; skew 500 ppm
I20260925 05:12:03.425509 28987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:12:03.433823 28987 webserver.cc:460] Webserver started at http://127.8.215.1:37285/ using document root <none> and password file <none>
I20260925 05:12:03.434634 28987 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:03.434777 28987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:03.435137 28987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:03.438308 28987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data/instance:
uuid: "6c3a53c3543f43c9bd1d6e0373134701"
format_stamp: "Formatted at 2026-09-25 05:12:03 on dist-test-slave-wpdm"
server_key: "1f84d11e2cbc13e38821d901229289cf"
server_key_iv: "43b9cea1889d45b4af1ea05c4ba203d6"
server_key_version: "encryptionkey@0"
I20260925 05:12:03.439142 28987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance:
uuid: "6c3a53c3543f43c9bd1d6e0373134701"
format_stamp: "Formatted at 2026-09-25 05:12:03 on dist-test-slave-wpdm"
server_key: "1f84d11e2cbc13e38821d901229289cf"
server_key_iv: "43b9cea1889d45b4af1ea05c4ba203d6"
server_key_version: "encryptionkey@0"
I20260925 05:12:03.444463 28987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:12:03.448225 29001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:03.449811 28987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:03.449985 28987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal
uuid: "6c3a53c3543f43c9bd1d6e0373134701"
format_stamp: "Formatted at 2026-09-25 05:12:03 on dist-test-slave-wpdm"
server_key: "1f84d11e2cbc13e38821d901229289cf"
server_key_iv: "43b9cea1889d45b4af1ea05c4ba203d6"
server_key_version: "encryptionkey@0"
I20260925 05:12:03.450215 28987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:03.466724 28987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:03.467638 28987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:03.467931 28987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:03.469107 28987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:03.470695 28987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:03.470786 28987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:03.470933 28987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:03.471011 28987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:03.504750 28987 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.1:35079
I20260925 05:12:03.504851 29113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.1:35079 every 8 connection(s)
I20260925 05:12:03.506148 28987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:03.507777 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 28987
I20260925 05:12:03.507951 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal/instance
I20260925 05:12:03.508556 9052 external_mini_cluster.cc:1428] Setting key 35aefb34069639c9a20bf32b08b8a3e5
I20260925 05:12:03.511482 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.2:0
--local_ip_for_outbound_sockets=127.8.215.2
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38979
--builtin_ntp_servers=127.8.215.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:12:03.520642 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:03.509128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:56293 (local address 127.8.215.62:38979)
0925 05:12:03.509628 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:03.509632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:03.511156 (+ 1524us) server_negotiation.cc:407] Connection header received
0925 05:12:03.513108 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:03.513116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:03.513212 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:03.513337 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:03.514482 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.515641 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.517351 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.518123 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.518275 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:03.519615 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:03.519624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:03.519632 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:03.519660 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:03.519713 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:03.519740 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:03.519836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:03.520039 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 05:12:03.520155 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":269,"threads_started":1}
I20260925 05:12:03.521644 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:03.509835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38979 (local address 127.8.215.1:56293)
0925 05:12:03.510974 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:03.511027 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:03.512827 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:03.513545 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:03.513552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:03.513648 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:03.514306 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:03.514323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.515805 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.515808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:03.517203 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:03.517214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.518342 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.518347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:03.518547 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:03.519284 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:12:03.519286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:03.519327 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:12:03.519409 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:03.519838 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:03.519846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:03.519852 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:12:03.519996 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 05:12:03.520331 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":395,"threads_started":1}
I20260925 05:12:03.523373 29114 heartbeater.cc:347] Connected to a master server at 127.8.215.62:38979
I20260925 05:12:03.523715 29114 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:03.524675 29114 heartbeater.cc:511] Master 127.8.215.62:38979 requested a full tablet report, sending...
I20260925 05:12:03.526813 28928 ts_manager.cc:194] Registered new tserver with Master: 6c3a53c3543f43c9bd1d6e0373134701 (127.8.215.1:35079)
I20260925 05:12:03.528282 28928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.1:56293
W20260925 05:12:03.851565 29118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:03.851953 29118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:03.852129 29118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:03.859943 29118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:03.860107 29118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:03.860293 29118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.2
I20260925 05:12:03.870038 29118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.215.2
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.2
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:03.871465 29118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:03.873438 29118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:03.887300 29126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:03.887398 29124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:03.887346 29123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:03.890400 29118 server_base.cc:1034] running on GCE node
I20260925 05:12:03.891065 29118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:03.892086 29118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:03.893304 29118 hybrid_clock.cc:648] HybridClock initialized: now 1790313123893235 us; error 68 us; skew 500 ppm
I20260925 05:12:03.893673 29118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:12:03.896018 29118 webserver.cc:460] Webserver started at http://127.8.215.2:35213/ using document root <none> and password file <none>
I20260925 05:12:03.896857 29118 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:03.896978 29118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:03.897313 29118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:03.900223 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data/instance:
uuid: "33351b4665b744a68269914e77e184cd"
format_stamp: "Formatted at 2026-09-25 05:12:03 on dist-test-slave-wpdm"
server_key: "52a91362e800fa5605de9c4982d2f749"
server_key_iv: "5112a4bc4af1b17ddc49da7c4904ef78"
server_key_version: "encryptionkey@0"
I20260925 05:12:03.901108 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance:
uuid: "33351b4665b744a68269914e77e184cd"
format_stamp: "Formatted at 2026-09-25 05:12:03 on dist-test-slave-wpdm"
server_key: "52a91362e800fa5605de9c4982d2f749"
server_key_iv: "5112a4bc4af1b17ddc49da7c4904ef78"
server_key_version: "encryptionkey@0"
I20260925 05:12:03.906401 29118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 05:12:03.910023 29132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:03.911482 29118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:12:03.911621 29118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal
uuid: "33351b4665b744a68269914e77e184cd"
format_stamp: "Formatted at 2026-09-25 05:12:03 on dist-test-slave-wpdm"
server_key: "52a91362e800fa5605de9c4982d2f749"
server_key_iv: "5112a4bc4af1b17ddc49da7c4904ef78"
server_key_version: "encryptionkey@0"
I20260925 05:12:03.911880 29118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:03.937055 29118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:03.938086 29118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:03.938371 29118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:03.939494 29118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:03.941347 29118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:03.941450 29118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:03.941552 29118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:03.941632 29118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:03.976052 29118 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.2:36395
I20260925 05:12:03.976101 29244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.2:36395 every 8 connection(s)
I20260925 05:12:03.977413 29118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/data/info.pb
I20260925 05:12:03.981137 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29118
I20260925 05:12:03.981346 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal/instance
I20260925 05:12:03.981964 9052 external_mini_cluster.cc:1428] Setting key 78833948c22ad07c2ff4b663a8f8dd63
I20260925 05:12:03.985208 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.215.3:0
--local_ip_for_outbound_sockets=127.8.215.3
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38979
--builtin_ntp_servers=127.8.215.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:12:03.992046 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:03.979948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.2:43997 (local address 127.8.215.62:38979)
0925 05:12:03.980122 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:03.980127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:03.981731 (+ 1604us) server_negotiation.cc:407] Connection header received
0925 05:12:03.984022 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:03.984028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:03.984110 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:03.984200 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:03.985787 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.986509 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.989135 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.989537 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.989634 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:03.991300 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:03.991304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:03.991308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:03.991326 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:03.991360 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:03.991371 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:03.991438 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:03.991731 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:12:03.991815 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:12:03.993662 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:03.980429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38979 (local address 127.8.215.2:43997)
0925 05:12:03.981542 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:03.981591 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:03.983757 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:03.984515 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:03.984528 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:03.984690 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:03.985592 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:03.985630 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.986695 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.986704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:03.988993 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:03.989011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:03.989773 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:03.989783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:03.990122 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:03.990901 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:12:03.990906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:03.990963 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:12:03.991082 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:03.991470 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:03.991482 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:03.991493 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:12:03.991690 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 05:12:03.992157 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":388,"threads_started":1}
I20260925 05:12:03.995187 29245 heartbeater.cc:347] Connected to a master server at 127.8.215.62:38979
I20260925 05:12:03.995519 29245 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:03.996476 29245 heartbeater.cc:511] Master 127.8.215.62:38979 requested a full tablet report, sending...
I20260925 05:12:03.998199 28928 ts_manager.cc:194] Registered new tserver with Master: 33351b4665b744a68269914e77e184cd (127.8.215.2:36395)
I20260925 05:12:03.999017 28928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.2:43997
W20260925 05:12:04.327684 29248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:04.327980 29248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:04.328140 29248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:04.335621 29248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:04.335737 29248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:04.335914 29248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.215.3
I20260925 05:12:04.345180 29248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.215.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.215.3
--webserver_port=0
--tserver_master_addrs=127.8.215.62:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.215.3
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:04.346561 29248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:04.348404 29248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:04.362092 29253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:04.362051 29256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:04.362064 29254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:04.363425 29248 server_base.cc:1034] running on GCE node
I20260925 05:12:04.364053 29248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:04.365123 29248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:04.366328 29248 hybrid_clock.cc:648] HybridClock initialized: now 1790313124366275 us; error 39 us; skew 500 ppm
I20260925 05:12:04.366671 29248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:12:04.369202 29248 webserver.cc:460] Webserver started at http://127.8.215.3:45663/ using document root <none> and password file <none>
I20260925 05:12:04.369925 29248 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:04.370040 29248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:04.370383 29248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:04.373634 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data/instance:
uuid: "d2c0193d3f1f4108bd1666b06dd92c0a"
format_stamp: "Formatted at 2026-09-25 05:12:04 on dist-test-slave-wpdm"
server_key: "f13d756ae25054077963cdb9ba6f6ecc"
server_key_iv: "d98778eaabf1a48c1898080351b7b7dc"
server_key_version: "encryptionkey@0"
I20260925 05:12:04.374495 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance:
uuid: "d2c0193d3f1f4108bd1666b06dd92c0a"
format_stamp: "Formatted at 2026-09-25 05:12:04 on dist-test-slave-wpdm"
server_key: "f13d756ae25054077963cdb9ba6f6ecc"
server_key_iv: "d98778eaabf1a48c1898080351b7b7dc"
server_key_version: "encryptionkey@0"
I20260925 05:12:04.379827 29248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:12:04.383764 29262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:04.385366 29248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:12:04.385548 29248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal
uuid: "d2c0193d3f1f4108bd1666b06dd92c0a"
format_stamp: "Formatted at 2026-09-25 05:12:04 on dist-test-slave-wpdm"
server_key: "f13d756ae25054077963cdb9ba6f6ecc"
server_key_iv: "d98778eaabf1a48c1898080351b7b7dc"
server_key_version: "encryptionkey@0"
I20260925 05:12:04.385771 29248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:04.409016 29248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:04.409945 29248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:04.410189 29248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:04.411108 29248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:04.412533 29248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:04.412650 29248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:04.412778 29248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:04.412861 29248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:04.445264 29248 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.3:45651
I20260925 05:12:04.445369 29374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.3:45651 every 8 connection(s)
I20260925 05:12:04.446655 29248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:04.456707 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29248
I20260925 05:12:04.456919 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal/instance
I20260925 05:12:04.457367 9052 external_mini_cluster.cc:1428] Setting key db175f40c87a7e2d5349e793904544e6
I20260925 05:12:04.459180 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.449308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.3:59247 (local address 127.8.215.62:38979)
0925 05:12:04.449488 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.449492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.450870 (+ 1378us) server_negotiation.cc:407] Connection header received
0925 05:12:04.452539 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.452544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.452629 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.452762 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:04.453794 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.454644 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.456124 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.456706 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.456877 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.458207 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:04.458223 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:04.458227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:04.458247 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:04.458279 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:04.458290 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:04.458402 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.458685 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.458782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:12:04.460126 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.449813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38979 (local address 127.8.215.3:59247)
0925 05:12:04.450709 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.450750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:04.452308 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:04.452969 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:04.452977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:04.453069 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:04.453622 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.453639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.454786 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.454790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.456004 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.456015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.456956 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.456961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.457209 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.457916 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:12:04.457919 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:04.457959 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:12:04.458037 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:04.458404 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:04.458418 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:04.458428 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:12:04.458641 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 05:12:04.458971 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
I20260925 05:12:04.461473 29375 heartbeater.cc:347] Connected to a master server at 127.8.215.62:38979
I20260925 05:12:04.461771 29375 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:04.462468 29375 heartbeater.cc:511] Master 127.8.215.62:38979 requested a full tablet report, sending...
I20260925 05:12:04.463814 28928 ts_manager.cc:194] Registered new tserver with Master: d2c0193d3f1f4108bd1666b06dd92c0a (127.8.215.3:45651)
I20260925 05:12:04.464640 28928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.215.3:59247
I20260925 05:12:04.472826 9052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:12:04.482355 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.476863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.8.215.62:38979)
0925 05:12:04.477097 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.477107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.477450 (+ 343us) server_negotiation.cc:407] Connection header received
0925 05:12:04.477577 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.477581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.477652 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.477740 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:04.478183 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.478862 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.480141 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.480532 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.480634 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.481710 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:04.481714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:04.481727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:04.481742 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:04.481771 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:04.481782 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:04.481850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.481963 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.482034 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:12:04.485977 28928 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:12:04.492715 28928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:12:04.522912 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.504488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36395 (local address 127.0.0.1:60412)
0925 05:12:04.505516 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.505559 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:04.505749 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:04.506888 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:04.506899 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:04.506941 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:04.508641 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.508660 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.512511 (+ 3851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.512519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.517036 (+ 4517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.517057 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.520045 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.520055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.521829 (+ 1774us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.521924 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 05:12:04.522078 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 05:12:04.522368 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":37,"thread_start_us":809,"threads_started":1}
I20260925 05:12:04.524026 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.505110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.8.215.2:36395)
0925 05:12:04.505616 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.505624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.505656 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:12:04.505990 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.505999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.506290 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.506527 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:04.508934 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.512311 (+ 3377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.517331 (+ 5020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.519367 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.519538 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.519864 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.523269 (+ 3405us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.523544 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":806,"threads_started":1}
I20260925 05:12:04.525938 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.506082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:45651 (local address 127.0.0.1:51656)
0925 05:12:04.513789 (+ 7707us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.513805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:04.513960 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:04.515092 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:04.515100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:04.515197 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:04.515877 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.515894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.518706 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.518714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.521755 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.521775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.524634 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.524643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.525054 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.525106 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 05:12:04.525227 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 05:12:04.525416 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7575,"mutex_wait_us":41,"thread_start_us":7444,"threads_started":1}
I20260925 05:12:04.526062 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.506514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.8.215.3:45651)
0925 05:12:04.513443 (+ 6929us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.513453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.514110 (+ 657us) server_negotiation.cc:407] Connection header received
0925 05:12:04.514226 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.514234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.514503 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.514742 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:04.516129 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.518489 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.522032 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.523982 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.524137 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.524455 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.525317 (+ 862us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.525560 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6750,"thread_start_us":6607,"threads_started":1}
I20260925 05:12:04.531302 29114 heartbeater.cc:503] Master 127.8.215.62:38979 was elected leader, sending a full tablet report...
I20260925 05:12:04.532334 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.513908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50438)
0925 05:12:04.524464 (+ 10556us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.524489 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:04.524621 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:04.527116 (+ 2495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:04.527120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:04.527144 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:04.527325 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.527336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.529154 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.529158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.530238 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.530250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.531621 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.531628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.531827 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.531867 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:12:04.531938 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:12:04.532065 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10414,"thread_start_us":13750,"threads_started":1}
I20260925 05:12:04.531785 29310 tablet_service.cc:1467] Processing CreateTablet for tablet cbc312a133174db19c06b3d7a6c50713 (DEFAULT_TABLE table=test-table [id=35412bc4e45f40049824faee3905d34f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:04.531738 29180 tablet_service.cc:1467] Processing CreateTablet for tablet cbc312a133174db19c06b3d7a6c50713 (DEFAULT_TABLE table=test-table [id=35412bc4e45f40049824faee3905d34f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:04.533273 29310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc312a133174db19c06b3d7a6c50713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:04.533281 29180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc312a133174db19c06b3d7a6c50713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:04.533536 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.514239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.8.215.1:35079)
0925 05:12:04.526617 (+ 12378us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.526624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.526648 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:12:04.526723 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.526728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.526918 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.527147 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:04.527485 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.528997 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.530397 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.531431 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.532782 (+ 1351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.533058 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.533117 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.533226 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12220,"thread_start_us":13101,"threads_started":1}
I20260925 05:12:04.536830 29049 tablet_service.cc:1467] Processing CreateTablet for tablet cbc312a133174db19c06b3d7a6c50713 (DEFAULT_TABLE table=test-table [id=35412bc4e45f40049824faee3905d34f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:04.538019 29049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc312a133174db19c06b3d7a6c50713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:04.548576 29393 tablet_bootstrap.cc:492] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Bootstrap starting.
I20260925 05:12:04.549603 29394 tablet_bootstrap.cc:492] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Bootstrap starting.
I20260925 05:12:04.550666 29395 tablet_bootstrap.cc:492] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Bootstrap starting.
I20260925 05:12:04.553774 29395 tablet_bootstrap.cc:654] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:04.554322 29393 tablet_bootstrap.cc:654] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:04.554346 29394 tablet_bootstrap.cc:654] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:04.555116 29395 log.cc:824] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:04.555593 29393 log.cc:824] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:04.556699 29394 log.cc:824] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:04.557857 29395 tablet_bootstrap.cc:492] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: No bootstrap required, opened a new log
I20260925 05:12:04.558118 29395 ts_tablet_manager.cc:1397] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:12:04.559149 29394 tablet_bootstrap.cc:492] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: No bootstrap required, opened a new log
I20260925 05:12:04.559393 29394 ts_tablet_manager.cc:1397] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 05:12:04.563130 29393 tablet_bootstrap.cc:492] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: No bootstrap required, opened a new log
I20260925 05:12:04.563387 29393 ts_tablet_manager.cc:1397] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260925 05:12:04.565410 29395 raft_consensus.cc:348] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.565789 29395 raft_consensus.cc:374] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:04.565917 29395 raft_consensus.cc:729] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3a53c3543f43c9bd1d6e0373134701, State: Initialized, Role: FOLLOWER
I20260925 05:12:04.566692 29395 consensus_queue.cc:260] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.568974 29394 raft_consensus.cc:348] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.569406 29394 raft_consensus.cc:374] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:04.569510 29394 raft_consensus.cc:729] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33351b4665b744a68269914e77e184cd, State: Initialized, Role: FOLLOWER
I20260925 05:12:04.570122 29395 ts_tablet_manager.cc:1428] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260925 05:12:04.570147 29394 consensus_queue.cc:260] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.571235 29245 heartbeater.cc:503] Master 127.8.215.62:38979 was elected leader, sending a full tablet report...
I20260925 05:12:04.571744 29394 ts_tablet_manager.cc:1428] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 05:12:04.576350 29393 raft_consensus.cc:348] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.576664 29393 raft_consensus.cc:374] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:04.576785 29393 raft_consensus.cc:729] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0193d3f1f4108bd1666b06dd92c0a, State: Initialized, Role: FOLLOWER
I20260925 05:12:04.577268 29393 consensus_queue.cc:260] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.578130 29375 heartbeater.cc:503] Master 127.8.215.62:38979 was elected leader, sending a full tablet report...
I20260925 05:12:04.578548 29393 ts_tablet_manager.cc:1428] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 05:12:04.611155 29399 raft_consensus.cc:482] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:04.611439 29399 raft_consensus.cc:504] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.613101 29399 leader_election.cc:288] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33351b4665b744a68269914e77e184cd (127.8.215.2:36395), d2c0193d3f1f4108bd1666b06dd92c0a (127.8.215.3:45651)
I20260925 05:12:04.620242 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.613426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.2:36395 (local address 127.8.215.1:37395)
0925 05:12:04.614109 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.614128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:04.614279 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:04.614827 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:04.614832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:04.614866 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:04.615219 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.615231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.616496 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.616501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.618189 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.618200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.619431 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.619437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.619603 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.619639 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:12:04.619753 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:12:04.619905 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":22,"thread_start_us":368,"threads_started":1}
I20260925 05:12:04.620249 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.613473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:37395 (local address 127.8.215.2:36395)
0925 05:12:04.613652 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.613692 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.614252 (+ 560us) server_negotiation.cc:407] Connection header received
0925 05:12:04.614500 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.614506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.614603 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.614705 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:04.615364 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.616377 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.618386 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.619258 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.619351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.619413 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.619793 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.619946 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:12:04.620678 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.614494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.215.1:54353 (local address 127.8.215.3:45651)
0925 05:12:04.614738 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.614744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.614999 (+ 255us) server_negotiation.cc:407] Connection header received
0925 05:12:04.615125 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.615133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.615212 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.615320 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:04.615951 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.616918 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.618780 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.619706 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.619817 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.619930 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.620274 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.620429 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 05:12:04.620970 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.614313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.3:45651 (local address 127.8.215.1:54353)
0925 05:12:04.614896 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.614910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:04.615022 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:04.615355 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:04.615359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:04.615384 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:04.615818 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.615832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.617061 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.617065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.618586 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:04.618611 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.619913 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.619920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:04.620090 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.620123 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:12:04.620221 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:12:04.620592 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":22,"thread_start_us":306,"threads_started":1}
I20260925 05:12:04.621575 29200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc312a133174db19c06b3d7a6c50713" candidate_uuid: "6c3a53c3543f43c9bd1d6e0373134701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33351b4665b744a68269914e77e184cd" is_pre_election: true
I20260925 05:12:04.621917 29330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc312a133174db19c06b3d7a6c50713" candidate_uuid: "6c3a53c3543f43c9bd1d6e0373134701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" is_pre_election: true
I20260925 05:12:04.622071 29200 raft_consensus.cc:2454] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c3a53c3543f43c9bd1d6e0373134701 in term 0.
I20260925 05:12:04.622409 29330 raft_consensus.cc:2454] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c3a53c3543f43c9bd1d6e0373134701 in term 0.
I20260925 05:12:04.622749 29004 leader_election.cc:302] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33351b4665b744a68269914e77e184cd, 6c3a53c3543f43c9bd1d6e0373134701; no voters:
I20260925 05:12:04.623180 29399 raft_consensus.cc:2781] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:12:04.623342 29399 raft_consensus.cc:482] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:12:04.623453 29399 raft_consensus.cc:3037] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:04.625695 29399 raft_consensus.cc:504] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.626263 29399 leader_election.cc:288] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [CANDIDATE]: Term 1 election: Requested vote from peers 33351b4665b744a68269914e77e184cd (127.8.215.2:36395), d2c0193d3f1f4108bd1666b06dd92c0a (127.8.215.3:45651)
I20260925 05:12:04.626709 29200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc312a133174db19c06b3d7a6c50713" candidate_uuid: "6c3a53c3543f43c9bd1d6e0373134701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33351b4665b744a68269914e77e184cd"
I20260925 05:12:04.626886 29330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc312a133174db19c06b3d7a6c50713" candidate_uuid: "6c3a53c3543f43c9bd1d6e0373134701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a"
I20260925 05:12:04.626952 29200 raft_consensus.cc:3037] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:04.627115 29330 raft_consensus.cc:3037] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:04.629576 29330 raft_consensus.cc:2454] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3a53c3543f43c9bd1d6e0373134701 in term 1.
I20260925 05:12:04.629591 29200 raft_consensus.cc:2454] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3a53c3543f43c9bd1d6e0373134701 in term 1.
I20260925 05:12:04.629978 29004 leader_election.cc:302] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c3a53c3543f43c9bd1d6e0373134701, d2c0193d3f1f4108bd1666b06dd92c0a; no voters:
I20260925 05:12:04.630264 29399 raft_consensus.cc:2781] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:04.630714 29399 raft_consensus.cc:686] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3a53c3543f43c9bd1d6e0373134701, State: Running, Role: LEADER
I20260925 05:12:04.631173 29399 consensus_queue.cc:237] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } }
I20260925 05:12:04.635919 28928 catalog_manager.cc:5495] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c3a53c3543f43c9bd1d6e0373134701 (127.8.215.1). New cstate: current_term: 1 leader_uuid: "6c3a53c3543f43c9bd1d6e0373134701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c3a53c3543f43c9bd1d6e0373134701" member_type: VOTER last_known_addr { host: "127.8.215.1" port: 35079 } health_report { overall_health: HEALTHY } } }
I20260925 05:12:04.669271 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.664244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.8.215.1:35079)
0925 05:12:04.664429 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.664434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.664451 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:12:04.664563 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.664567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.664666 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.664800 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:04.665130 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.665796 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.667265 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.667669 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.667766 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.668186 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:04.668686 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:04.668874 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.668919 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.668979 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:12:04.677568 29330 raft_consensus.cc:1261] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 1 FOLLOWER]: Refusing update from remote peer 6c3a53c3543f43c9bd1d6e0373134701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:04.677600 29200 raft_consensus.cc:1261] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 1 FOLLOWER]: Refusing update from remote peer 6c3a53c3543f43c9bd1d6e0373134701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:04.678503 29404 consensus_queue.cc:1035] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33351b4665b744a68269914e77e184cd" member_type: VOTER last_known_addr { host: "127.8.215.2" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:04.678877 29399 consensus_queue.cc:1035] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c0193d3f1f4108bd1666b06dd92c0a" member_type: VOTER last_known_addr { host: "127.8.215.3" port: 45651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:04.705317 28927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:04.705552 28927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:04.709380 28927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89881b4addad47a3918abcd8b69b001d: Sending DeleteTablet for 3 replicas of tablet cbc312a133174db19c06b3d7a6c50713
I20260925 05:12:04.710588 29180 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc312a133174db19c06b3d7a6c50713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:04 UTC) from {username='slave'} at 127.0.0.1:60412
I20260925 05:12:04.710958 29049 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc312a133174db19c06b3d7a6c50713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:04 UTC) from {username='slave'} at 127.0.0.1:50438
I20260925 05:12:04.711246 29310 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc312a133174db19c06b3d7a6c50713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:04 UTC) from {username='slave'} at 127.0.0.1:51656
I20260925 05:12:04.713735 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28987
I20260925 05:12:04.722045 29418 tablet_replica.cc:330] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: stopping tablet replica
I20260925 05:12:04.722829 29418 raft_consensus.cc:2229] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:12:04.723939 29418 raft_consensus.cc:2258] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:04.724766 29419 tablet_replica.cc:330] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: stopping tablet replica
I20260925 05:12:04.725375 29419 raft_consensus.cc:2229] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:04.725962 29417 tablet_replica.cc:330] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: stopping tablet replica
I20260925 05:12:04.726557 29417 raft_consensus.cc:2229] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:04.727124 29417 raft_consensus.cc:2258] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:04.727903 29418 ts_tablet_manager.cc:1905] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:04.729346 29417 ts_tablet_manager.cc:1905] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:04.735536 29418 ts_tablet_manager.cc:1918] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:04.736189 29417 ts_tablet_manager.cc:1918] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:04.736245 29419 raft_consensus.cc:2258] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:04.736460 29417 log.cc:1196] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-1/wal/wals/cbc312a133174db19c06b3d7a6c50713
I20260925 05:12:04.737028 29418 log.cc:1196] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-0/wal/wals/cbc312a133174db19c06b3d7a6c50713
I20260925 05:12:04.737144 29417 ts_tablet_manager.cc:1939] T cbc312a133174db19c06b3d7a6c50713 P 33351b4665b744a68269914e77e184cd: Deleting consensus metadata
I20260925 05:12:04.738538 29419 ts_tablet_manager.cc:1905] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:04.739032 28914 catalog_manager.cc:4841] TS 33351b4665b744a68269914e77e184cd (127.8.215.2:36395): tablet cbc312a133174db19c06b3d7a6c50713 (table test-table [id=35412bc4e45f40049824faee3905d34f]) successfully deleted
I20260925 05:12:04.738260 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.715119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.8.215.1:35079)
0925 05:12:04.715295 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.715301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.729286 (+ 13985us) server_negotiation.cc:407] Connection header received
0925 05:12:04.729691 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.729697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.729805 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.729887 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:04.730626 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.731867 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.733973 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.734679 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.734785 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.736320 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:04.736333 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:04.736351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:04.737292 (+ 941us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:04.737360 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:04.737387 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:04.737477 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.737737 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.737847 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:12:04.740139 29109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:04.745999 29419 ts_tablet_manager.cc:1918] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:04.746173 29419 log.cc:1196] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Deleting WAL directory at /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312801912195-9052-0/minicluster-data/ts-2/wal/wals/cbc312a133174db19c06b3d7a6c50713
I20260925 05:12:04.746603 29419 ts_tablet_manager.cc:1939] T cbc312a133174db19c06b3d7a6c50713 P d2c0193d3f1f4108bd1666b06dd92c0a: Deleting consensus metadata
I20260925 05:12:04.748193 28914 catalog_manager.cc:4841] TS d2c0193d3f1f4108bd1666b06dd92c0a (127.8.215.3:45651): tablet cbc312a133174db19c06b3d7a6c50713 (table test-table [id=35412bc4e45f40049824faee3905d34f]) successfully deleted
I20260925 05:12:04.835783 29418 ts_tablet_manager.cc:1939] T cbc312a133174db19c06b3d7a6c50713 P 6c3a53c3543f43c9bd1d6e0373134701: Deleting consensus metadata
I20260925 05:12:04.836684 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28987
W20260925 05:12:04.854974 28914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.215.1:35079 (error 108)
I20260925 05:12:04.855623 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29118
I20260925 05:12:04.857252 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.856805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50450)
0925 05:12:04.857017 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.857106 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 05:12:04.857496 28914 catalog_manager.cc:4584] TS 6c3a53c3543f43c9bd1d6e0373134701 (127.8.215.1:35079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbc312a133174db19c06b3d7a6c50713: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
I20260925 05:12:04.862470 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.856473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.8.215.2:36395)
0925 05:12:04.856640 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.856644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.856661 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:12:04.856864 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.856869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.856961 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.857077 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:04.857557 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.858378 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.859611 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.860065 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.860174 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.861214 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:04.861222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:04.861236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:04.861872 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:04.861920 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:04.861941 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:04.862040 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.862144 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.862236 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:12:04.863111 29240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:04.911361 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.910809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50454)
0925 05:12:04.911055 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.911164 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 05:12:04.912472 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.912075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50460)
0925 05:12:04.912228 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.912309 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 05:12:04.956552 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29118
I20260925 05:12:04.973023 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.972562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50464)
0925 05:12:04.972789 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.972874 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 05:12:04.974119 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.973689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50474)
0925 05:12:04.973865 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:04.973930 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:12:04.974464 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29248
I20260925 05:12:04.980636 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:04.975184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.8.215.3:45651)
0925 05:12:04.975349 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:04.975354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:04.975397 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:12:04.975615 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:04.975619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:04.975706 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:04.975791 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:04.976129 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.976920 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.978124 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:04.978500 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:04.978588 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:04.979506 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:04.979514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:04.979528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:04.980093 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:04.980139 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:04.980158 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:04.980251 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:04.980346 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:12:04.980425 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:12:04.981285 29370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:05.054885 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:05.054396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50478)
0925 05:12:05.054646 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:05.054736 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 05:12:05.056180 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:05.055753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.1:35079 (local address 127.0.0.1:50492)
0925 05:12:05.055962 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:05.056042 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 05:12:05.068543 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29248
I20260925 05:12:05.086535 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 28897
I20260925 05:12:05.087623 28958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:05.195299 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 28897
2026-09-25T05:12:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2804 ms)
[----------] 6 tests from EncryptionPolicyTest (22034 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:05 dist-test-slave-wpdm krb5kdc[29428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:05 dist-test-slave-wpdm krb5kdc[29428](info): set up 2 sockets
Sep 25 05:12:05 dist-test-slave-wpdm krb5kdc[29428](info): commencing operation
krb5kdc: starting...
W20260925 05:12:06.321671 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:06 dist-test-slave-wpdm krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:06.620669 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:06.970911 29441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:06.971213 29441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:06.971374 29441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:06.979032 29441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:06.979153 29441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:06.979245 29441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:06.979357 29441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:06.979417 29441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:06.989534 29441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:06.990899 29441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:06.992236 29441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:07.004408 29446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:07.004441 29449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:07.004395 29447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:07.005461 29441 server_base.cc:1034] running on GCE node
I20260925 05:12:07.006253 29441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:07.007692 29441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:07.008927 29441 hybrid_clock.cc:648] HybridClock initialized: now 1790313127008892 us; error 62 us; skew 500 ppm
I20260925 05:12:07.009306 29441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:07 dist-test-slave-wpdm krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:07.012411 29441 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:07.014029 29441 webserver.cc:460] Webserver started at http://127.0.0.1:45399/ using document root <none> and password file <none>
I20260925 05:12:07.014739 29441 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:07.014889 29441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:07.015210 29441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:07.018419 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "fa4549dc14df40559282ff12716def5a"
format_stamp: "Formatted at 2026-09-25 05:12:07 on dist-test-slave-wpdm"
server_key: "43af82fb0c1154618fd892364baf7cf3"
server_key_iv: "1812904c613e1589f466fcdcca9f4b06"
server_key_version: "encryptionkey@0"
I20260925 05:12:07.019209 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "fa4549dc14df40559282ff12716def5a"
format_stamp: "Formatted at 2026-09-25 05:12:07 on dist-test-slave-wpdm"
server_key: "43af82fb0c1154618fd892364baf7cf3"
server_key_iv: "1812904c613e1589f466fcdcca9f4b06"
server_key_version: "encryptionkey@0"
I20260925 05:12:07.024564 29441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:12:07.028141 29456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:07.029507 29441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:07.029726 29441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "fa4549dc14df40559282ff12716def5a"
format_stamp: "Formatted at 2026-09-25 05:12:07 on dist-test-slave-wpdm"
server_key: "43af82fb0c1154618fd892364baf7cf3"
server_key_iv: "1812904c613e1589f466fcdcca9f4b06"
server_key_version: "encryptionkey@0"
I20260925 05:12:07.029951 29441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:07.044070 29441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:07.048242 29441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:07.048539 29441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:07.066797 29441 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37871
I20260925 05:12:07.066828 29507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37871 every 8 connection(s)
I20260925 05:12:07.068226 29441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:07.072257 29508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:07.078619 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29441
I20260925 05:12:07.078881 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:07.079505 9052 external_mini_cluster.cc:1428] Setting key 6985a8d1263b7e4ba5f2b81c618556d9
I20260925 05:12:07.083163 29508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a: Bootstrap starting.
I20260925 05:12:07.086966 29508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:07.088068 29508 log.cc:824] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:07.091324 29508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a: No bootstrap required, opened a new log
Sep 25 05:12:07 dist-test-slave-wpdm krb5kdc[29428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:12:07.100138 29508 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4549dc14df40559282ff12716def5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260925 05:12:07.100549 29508 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:07.100693 29508 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4549dc14df40559282ff12716def5a, State: Initialized, Role: FOLLOWER
I20260925 05:12:07.101481 29508 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4549dc14df40559282ff12716def5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260925 05:12:07.101804 29508 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:07.101985 29508 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:07.102200 29508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:07.103773 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:07.081484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40861 (local address 127.0.0.1:37871)
0925 05:12:07.082275 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:07.082287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:07.082330 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:12:07.084103 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:07.084127 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:07.084537 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:07.084929 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:07.086792 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:07.088229 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:07.089974 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:07.090650 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:07.090795 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:07.095607 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:07.095616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:07.095632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:07.095672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:07.099095 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:07.099983 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:07.099990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:07.099997 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:07.100170 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:07.100711 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:07.100716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:07.100720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:07.101452 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:07.101821 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:07.102335 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 05:12:07.102656 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":388,"threads_started":1}
I20260925 05:12:07.104883 29508 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4549dc14df40559282ff12716def5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260925 05:12:07.105438 29508 leader_election.cc:302] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4549dc14df40559282ff12716def5a; no voters:
I20260925 05:12:07.106140 29508 leader_election.cc:288] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:07.106273 29513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:07.107398 29513 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [term 1 LEADER]: Becoming Leader. State: Replica: fa4549dc14df40559282ff12716def5a, State: Running, Role: LEADER
I20260925 05:12:07.108023 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:07.108047 29513 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4549dc14df40559282ff12716def5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260925 05:12:07.108661 29508 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:07.111155 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4549dc14df40559282ff12716def5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4549dc14df40559282ff12716def5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } } }
I20260925 05:12:07.111605 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:07.115536 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4549dc14df40559282ff12716def5a. Latest consensus state: current_term: 1 leader_uuid: "fa4549dc14df40559282ff12716def5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4549dc14df40559282ff12716def5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } } }
I20260925 05:12:07.115995 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:07.117611 29519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:07.128556 29519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:07.145208 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:07.120792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51839 (local address 127.0.0.1:37871)
0925 05:12:07.121021 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:07.121030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:07.121706 (+ 676us) server_negotiation.cc:407] Connection header received
0925 05:12:07.121917 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:07.121922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:07.122001 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:07.122131 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:07.124562 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:07.126338 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:07.128949 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:07.130028 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:07.130269 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:07.134631 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:07.134642 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:07.134650 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:07.134705 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:07.139498 (+ 4793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:07.140888 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:07.140898 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:07.140904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:07.141112 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:07.141814 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:07.141822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:07.141828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:07.142374 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:07.142701 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:07.144378 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0925 05:12:07.144637 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:12:07.151185 29519 catalog_manager.cc:1347] Generated new cluster ID: 5197bdb528844cdd9416f8a8efedf442
I20260925 05:12:07.151412 29519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:07.208544 29519 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:07.210289 29519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:07.217809 29519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4549dc14df40559282ff12716def5a: Generated new TSK 0
I20260925 05:12:07.218679 29519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:07.234176 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29441
I20260925 05:12:07.235857 29503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:07.336019 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29441
W20260925 05:12:07.355216 29426 connection.cc:535] client connection to 127.0.0.1:37871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37871): Connection reset by peer (error 104)
2026-09-25T05:12:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2136 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:07 dist-test-slave-wpdm krb5kdc[29540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:07 dist-test-slave-wpdm krb5kdc[29540](info): set up 2 sockets
Sep 25 05:12:07 dist-test-slave-wpdm krb5kdc[29540](info): commencing operation
krb5kdc: starting...
W20260925 05:12:08.473675 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:08 dist-test-slave-wpdm krb5kdc[29540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:08.756194 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:09.106702 29553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:09.107107 29553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:09.107287 29553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:09.115680 29553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:09.115861 29553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:09.115975 29553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:09.116101 29553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:09.116187 29553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:09.126922 29553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:09.128232 29553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:09.129621 29553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:09.141887 29561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:09.141906 29558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:09.141935 29559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:09.143177 29553 server_base.cc:1034] running on GCE node
I20260925 05:12:09.143987 29553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:09.145565 29553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:09.146792 29553 hybrid_clock.cc:648] HybridClock initialized: now 1790313129146746 us; error 39 us; skew 500 ppm
I20260925 05:12:09.147152 29553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:09 dist-test-slave-wpdm krb5kdc[29540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:09.159258 29553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:09.161011 29553 webserver.cc:460] Webserver started at http://127.0.0.1:39439/ using document root <none> and password file <none>
I20260925 05:12:09.161729 29553 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:09.161877 29553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:09.162184 29553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:09.165233 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "ed1e0ae8fa2349e3878be613d3a02d56"
format_stamp: "Formatted at 2026-09-25 05:12:09 on dist-test-slave-wpdm"
server_key: "a052c89d4b474b73a84b6507ed34c93a"
server_key_iv: "277e0840c5540b20d7a0d6bf3bdc17b0"
server_key_version: "encryptionkey@0"
I20260925 05:12:09.166002 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "ed1e0ae8fa2349e3878be613d3a02d56"
format_stamp: "Formatted at 2026-09-25 05:12:09 on dist-test-slave-wpdm"
server_key: "a052c89d4b474b73a84b6507ed34c93a"
server_key_iv: "277e0840c5540b20d7a0d6bf3bdc17b0"
server_key_version: "encryptionkey@0"
I20260925 05:12:09.171026 29553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:12:09.174489 29568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:09.175786 29553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:12:09.175979 29553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "ed1e0ae8fa2349e3878be613d3a02d56"
format_stamp: "Formatted at 2026-09-25 05:12:09 on dist-test-slave-wpdm"
server_key: "a052c89d4b474b73a84b6507ed34c93a"
server_key_iv: "277e0840c5540b20d7a0d6bf3bdc17b0"
server_key_version: "encryptionkey@0"
I20260925 05:12:09.176187 29553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:09.191368 29553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:09.195741 29553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:09.196112 29553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:09.217324 29553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44577
I20260925 05:12:09.217355 29619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44577 every 8 connection(s)
I20260925 05:12:09.218739 29553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:09.222492 29620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:09.223711 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29553
I20260925 05:12:09.223901 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:09.224458 9052 external_mini_cluster.cc:1428] Setting key 8a78e2b7616d615982614f2dc71ee310
I20260925 05:12:09.236171 29620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56: Bootstrap starting.
I20260925 05:12:09.240029 29620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:09.240660 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:09.226592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50179 (local address 127.0.0.1:44577)
0925 05:12:09.227548 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:09.227565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:09.228019 (+ 454us) server_negotiation.cc:407] Connection header received
0925 05:12:09.230464 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:09.230493 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:09.231000 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:09.231445 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:09.232001 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:09.234115 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:09.235430 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:09.236457 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:09.236644 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:09.237903 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:09.237913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:09.237931 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:09.238642 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:09.238739 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:09.238771 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:09.238928 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:09.239176 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 05:12:09.239338 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":423,"threads_started":1}
I20260925 05:12:09.241330 29620 log.cc:824] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:09.244367 29620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56: No bootstrap required, opened a new log
I20260925 05:12:09.244623 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:09.255030 29620 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44577 } }
I20260925 05:12:09.255527 29620 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:09.255648 29620 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e0ae8fa2349e3878be613d3a02d56, State: Initialized, Role: FOLLOWER
I20260925 05:12:09.256333 29620 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44577 } }
I20260925 05:12:09.256672 29620 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:09.256716 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:09.249863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44155 (local address 127.0.0.1:44577)
0925 05:12:09.250047 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:09.250052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:09.250552 (+ 500us) server_negotiation.cc:407] Connection header received
0925 05:12:09.250735 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:09.250738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:09.250800 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:09.250881 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:09.251417 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:09.252167 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:09.254047 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:09.254479 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:09.254570 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:09.255897 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:09.255901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:09.255905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:09.255920 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:09.255982 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:09.255996 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:09.256122 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:09.256333 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:12:09.256385 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:12:09.256870 29620 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:09.257058 29620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:09.259708 29620 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44577 } }
I20260925 05:12:09.260226 29620 leader_election.cc:302] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed1e0ae8fa2349e3878be613d3a02d56; no voters:
I20260925 05:12:09.260854 29620 leader_election.cc:288] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:09.260977 29630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:09.261718 29630 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [term 1 LEADER]: Becoming Leader. State: Replica: ed1e0ae8fa2349e3878be613d3a02d56, State: Running, Role: LEADER
I20260925 05:12:09.262252 29630 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44577 } }
I20260925 05:12:09.263065 29620 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:09.265332 29631 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44577 } } }
I20260925 05:12:09.265753 29631 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:09.270658 29632 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed1e0ae8fa2349e3878be613d3a02d56. Latest consensus state: current_term: 1 leader_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed1e0ae8fa2349e3878be613d3a02d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44577 } } }
I20260925 05:12:09.271047 29632 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:09.273306 29638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:09.278950 29638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:09.288537 29638 catalog_manager.cc:1347] Generated new cluster ID: c09528b1cbee4ac7966d2fdfbcf25978
I20260925 05:12:09.288672 29638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:09.308144 29638 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:09.309960 29638 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:09.319018 29638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed1e0ae8fa2349e3878be613d3a02d56: Generated new TSK 0
I20260925 05:12:09.320080 29638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:09.331414 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29553
W20260925 05:12:09.332747 29615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50179
W20260925 05:12:09.333539 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:09.333741 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29553
W20260925 05:12:09.351405 29538 connection.cc:535] client connection to 127.0.0.1:44577 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44577): Connection reset by peer (error 104)
2026-09-25T05:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:09 dist-test-slave-wpdm krb5kdc[29651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:09 dist-test-slave-wpdm krb5kdc[29651](info): set up 2 sockets
Sep 25 05:12:09 dist-test-slave-wpdm krb5kdc[29651](info): commencing operation
krb5kdc: starting...
W20260925 05:12:10.445833 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:10 dist-test-slave-wpdm krb5kdc[29651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:10.717988 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:11.057333 29664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:11.057610 29664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:11.057771 29664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:11.065026 29664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:11.065140 29664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:11.065244 29664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:11.065343 29664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:11.065429 29664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:11.075376 29664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:11.076823 29664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:11.078145 29664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:11.090175 29669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:11.090196 29670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:11.090169 29672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:11.091344 29664 server_base.cc:1034] running on GCE node
I20260925 05:12:11.092093 29664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:11.093593 29664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:11.094787 29664 hybrid_clock.cc:648] HybridClock initialized: now 1790313131094723 us; error 50 us; skew 500 ppm
I20260925 05:12:11.095139 29664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:11 dist-test-slave-wpdm krb5kdc[29651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:11.098378 29664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:11.099805 29664 webserver.cc:460] Webserver started at http://127.0.0.1:38639/ using document root <none> and password file <none>
I20260925 05:12:11.100490 29664 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:11.100639 29664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:11.100983 29664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:11.104065 29664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "6cff7aca29ed422c97ac4bafecbd7cd9"
format_stamp: "Formatted at 2026-09-25 05:12:11 on dist-test-slave-wpdm"
server_key: "455c98bb09cd55ba98fe8b9dd4057499"
server_key_iv: "098182b3262241abef37076733041903"
server_key_version: "encryptionkey@0"
I20260925 05:12:11.104933 29664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "6cff7aca29ed422c97ac4bafecbd7cd9"
format_stamp: "Formatted at 2026-09-25 05:12:11 on dist-test-slave-wpdm"
server_key: "455c98bb09cd55ba98fe8b9dd4057499"
server_key_iv: "098182b3262241abef37076733041903"
server_key_version: "encryptionkey@0"
I20260925 05:12:11.110229 29664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:12:11.114030 29679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:11.115435 29664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:12:11.115626 29664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "6cff7aca29ed422c97ac4bafecbd7cd9"
format_stamp: "Formatted at 2026-09-25 05:12:11 on dist-test-slave-wpdm"
server_key: "455c98bb09cd55ba98fe8b9dd4057499"
server_key_iv: "098182b3262241abef37076733041903"
server_key_version: "encryptionkey@0"
I20260925 05:12:11.115835 29664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:11.121711 29664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:11.126102 29664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:11.126417 29664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:11.144954 29664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46587
I20260925 05:12:11.145042 29730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46587 every 8 connection(s)
I20260925 05:12:11.146401 29664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:11.150636 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:11.154452 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29664
I20260925 05:12:11.154644 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:11.155082 9052 external_mini_cluster.cc:1428] Setting key 6f76b29123e77f90b2d4a1b7fe2f5eb3
I20260925 05:12:11.163684 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:11.157012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36053 (local address 127.0.0.1:46587)
0925 05:12:11.157814 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:11.157829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:11.157877 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:12:11.159747 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:11.159781 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:11.160273 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:11.160760 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:11.161762 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:11.161770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:11.161786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:11.161809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:11.161877 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:11.161908 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:11.162140 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:11.162283 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 05:12:11.162466 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":328,"threads_started":1}
I20260925 05:12:11.164448 29731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9: Bootstrap starting.
I20260925 05:12:11.167412 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:11.167804 29731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:11.168887 29731 log.cc:824] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:11.172335 29731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9: No bootstrap required, opened a new log
I20260925 05:12:11.174598 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:11.172023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50959 (local address 127.0.0.1:46587)
0925 05:12:11.172227 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:11.172233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:11.172529 (+ 296us) server_negotiation.cc:407] Connection header received
0925 05:12:11.172708 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:11.172712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:11.172807 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:11.172894 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:11.173891 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:11.173895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:11.173898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:11.173912 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:11.173950 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:11.173965 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:11.174112 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:11.174208 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 05:12:11.174255 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:12:11.180985 29731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46587 } }
I20260925 05:12:11.181561 29731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:11.181681 29731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cff7aca29ed422c97ac4bafecbd7cd9, State: Initialized, Role: FOLLOWER
I20260925 05:12:11.182265 29731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46587 } }
I20260925 05:12:11.182483 29731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:11.182597 29731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:11.182730 29731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:11.184955 29731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46587 } }
I20260925 05:12:11.185437 29731 leader_election.cc:302] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cff7aca29ed422c97ac4bafecbd7cd9; no voters:
I20260925 05:12:11.186023 29731 leader_election.cc:288] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:11.186211 29741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:11.186919 29741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [term 1 LEADER]: Becoming Leader. State: Replica: 6cff7aca29ed422c97ac4bafecbd7cd9, State: Running, Role: LEADER
I20260925 05:12:11.187561 29741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46587 } }
I20260925 05:12:11.188489 29731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:11.191020 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46587 } } }
I20260925 05:12:11.191043 29743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cff7aca29ed422c97ac4bafecbd7cd9. Latest consensus state: current_term: 1 leader_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cff7aca29ed422c97ac4bafecbd7cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46587 } } }
I20260925 05:12:11.191596 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:11.191596 29743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:11.194106 29749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:11.199741 29749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:11.210310 29749 catalog_manager.cc:1347] Generated new cluster ID: 9b8115a58d00418fb941d800012ce528
I20260925 05:12:11.210454 29749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:11.239951 29749 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:11.240432 29749 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:11.253994 29749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cff7aca29ed422c97ac4bafecbd7cd9: Generated new TSK 0
I20260925 05:12:11.254900 29749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:11.261255 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29664
W20260925 05:12:11.262405 29726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36053
W20260925 05:12:11.263159 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:11.263310 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29664
2026-09-25T05:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:11 dist-test-slave-wpdm krb5kdc[29762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:11 dist-test-slave-wpdm krb5kdc[29762](info): set up 2 sockets
Sep 25 05:12:11 dist-test-slave-wpdm krb5kdc[29762](info): commencing operation
krb5kdc: starting...
W20260925 05:12:12.425719 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:12 dist-test-slave-wpdm krb5kdc[29762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:12.695901 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:13.032270 29775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:13.032534 29775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:13.032706 29775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:13.039841 29775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:13.039984 29775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:13.040086 29775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:13.040190 29775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:13.040275 29775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:13.049957 29775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:13.051241 29775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:13.052532 29775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:13.063982 29780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:13.064037 29781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:13.064008 29783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:13.064988 29775 server_base.cc:1034] running on GCE node
I20260925 05:12:13.065730 29775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:13.067070 29775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:13.068302 29775 hybrid_clock.cc:648] HybridClock initialized: now 1790313133068257 us; error 57 us; skew 500 ppm
I20260925 05:12:13.068639 29775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:13 dist-test-slave-wpdm krb5kdc[29762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:13.071681 29775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:13.073117 29775 webserver.cc:460] Webserver started at http://127.0.0.1:34421/ using document root <none> and password file <none>
I20260925 05:12:13.073822 29775 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:13.073963 29775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:13.074273 29775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:13.077219 29775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "86d0ed9baf42435187fad06432798d56"
format_stamp: "Formatted at 2026-09-25 05:12:13 on dist-test-slave-wpdm"
server_key: "0a40e7f305cf8670b3aaf4eafe976bd7"
server_key_iv: "4cbc0bd7291b54e49626708126bf9b54"
server_key_version: "encryptionkey@0"
I20260925 05:12:13.077976 29775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "86d0ed9baf42435187fad06432798d56"
format_stamp: "Formatted at 2026-09-25 05:12:13 on dist-test-slave-wpdm"
server_key: "0a40e7f305cf8670b3aaf4eafe976bd7"
server_key_iv: "4cbc0bd7291b54e49626708126bf9b54"
server_key_version: "encryptionkey@0"
I20260925 05:12:13.083004 29775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 05:12:13.086519 29790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:13.087818 29775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:12:13.087997 29775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "86d0ed9baf42435187fad06432798d56"
format_stamp: "Formatted at 2026-09-25 05:12:13 on dist-test-slave-wpdm"
server_key: "0a40e7f305cf8670b3aaf4eafe976bd7"
server_key_iv: "4cbc0bd7291b54e49626708126bf9b54"
server_key_version: "encryptionkey@0"
I20260925 05:12:13.088210 29775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:13.116333 29775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:13.120922 29775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:13.121213 29775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:13.139343 29775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46605
I20260925 05:12:13.139371 29841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46605 every 8 connection(s)
I20260925 05:12:13.140743 29775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:13.141978 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29775
I20260925 05:12:13.142194 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:13.142625 9052 external_mini_cluster.cc:1428] Setting key 206acdd92fe5ac5a9980dec0d4bd41fd
I20260925 05:12:13.145084 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:12:13 dist-test-slave-wpdm krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:12:13.157903 29842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56: Bootstrap starting.
I20260925 05:12:13.161844 29842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:13.163174 29842 log.cc:824] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:13.166405 29842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56: No bootstrap required, opened a new log
I20260925 05:12:13.167394 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:13.144473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37065 (local address 127.0.0.1:46605)
0925 05:12:13.145282 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:13.145297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:13.145345 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:12:13.147652 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:13.147679 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:13.148205 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:13.148646 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:13.150122 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:13.152161 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:13.153577 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:13.154638 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:13.154819 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:13.158935 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:13.158945 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:13.158961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:13.159005 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:13.162926 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:13.163667 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:13.163680 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:13.163695 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:13.163931 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:13.164329 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:13.164335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:13.164340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:13.165026 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:13.165319 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:13.165683 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 05:12:13.166103 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":337,"threads_started":1}
I20260925 05:12:13.171060 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:13.174530 29842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d0ed9baf42435187fad06432798d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260925 05:12:13.174891 29842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:13.174998 29842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d0ed9baf42435187fad06432798d56, State: Initialized, Role: FOLLOWER
I20260925 05:12:13.175699 29842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d0ed9baf42435187fad06432798d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260925 05:12:13.175992 29842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:13.176146 29842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:13.176350 29842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:13.179003 29842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d0ed9baf42435187fad06432798d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260925 05:12:13.179535 29842 leader_election.cc:302] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86d0ed9baf42435187fad06432798d56; no voters:
I20260925 05:12:13.180136 29842 leader_election.cc:288] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:13.180289 29852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:13.182116 29842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:13.183210 29852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [term 1 LEADER]: Becoming Leader. State: Replica: 86d0ed9baf42435187fad06432798d56, State: Running, Role: LEADER
I20260925 05:12:13.183938 29852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d0ed9baf42435187fad06432798d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260925 05:12:13.187731 29853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86d0ed9baf42435187fad06432798d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d0ed9baf42435187fad06432798d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } } }
I20260925 05:12:13.189127 29853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:13.187740 29854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d0ed9baf42435187fad06432798d56. Latest consensus state: current_term: 1 leader_uuid: "86d0ed9baf42435187fad06432798d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d0ed9baf42435187fad06432798d56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } } }
I20260925 05:12:13.189990 29854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:13.196990 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:13.175304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41657 (local address 127.0.0.1:46605)
0925 05:12:13.175467 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:13.175473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:13.176070 (+ 597us) server_negotiation.cc:407] Connection header received
0925 05:12:13.176258 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:13.176263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:13.176358 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:13.176470 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:13.177893 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:13.178961 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:13.180863 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:13.181851 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:13.181991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:13.185563 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:13.185569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:13.185574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:13.185621 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:13.191486 (+ 5865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:13.192756 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:13.192765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:13.192772 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:13.192979 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:13.193603 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:13.193611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:13.193617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:13.194102 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:13.195844 (+ 1742us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:13.196018 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:12:13.196338 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:12:13.198899 29863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:13.204209 29863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:13.213513 29863 catalog_manager.cc:1347] Generated new cluster ID: 6c38cf567ad645be84e07160b6cfe9c9
I20260925 05:12:13.213640 29863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:13.220644 29863 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:13.221899 29863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:13.236183 29863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86d0ed9baf42435187fad06432798d56: Generated new TSK 0
I20260925 05:12:13.237300 29863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:13.245424 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29775
I20260925 05:12:13.246436 29837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:13.336265 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29775
2026-09-25T05:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:13 dist-test-slave-wpdm krb5kdc[29874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:13 dist-test-slave-wpdm krb5kdc[29874](info): set up 2 sockets
Sep 25 05:12:13 dist-test-slave-wpdm krb5kdc[29874](info): commencing operation
krb5kdc: starting...
W20260925 05:12:14.454005 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:14 dist-test-slave-wpdm krb5kdc[29874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:14.731807 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:15.078472 29887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:15.078794 29887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:15.078984 29887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:15.086530 29887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:15.086719 29887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:15.086814 29887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:15.086920 29887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:15.087019 29887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:15.097553 29887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:15.098974 29887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:15.100387 29887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:15.112861 29892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:15.112920 29895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:15.112758 29893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:15.114007 29887 server_base.cc:1034] running on GCE node
I20260925 05:12:15.114874 29887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:15.116358 29887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:15.117614 29887 hybrid_clock.cc:648] HybridClock initialized: now 1790313135117567 us; error 54 us; skew 500 ppm
I20260925 05:12:15.118026 29887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:15 dist-test-slave-wpdm krb5kdc[29874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:15.127350 29887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:15.128975 29887 webserver.cc:460] Webserver started at http://127.0.0.1:38597/ using document root <none> and password file <none>
I20260925 05:12:15.129688 29887 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:15.129801 29887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:15.130098 29887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:15.133299 29887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "91a79a129fba4d749d9ca45bcb0ac986"
format_stamp: "Formatted at 2026-09-25 05:12:15 on dist-test-slave-wpdm"
server_key: "29b4fb8f2c83ac7f2967b5b6b587ba06"
server_key_iv: "88c6c111f52688c6b541d4031c30fbd9"
server_key_version: "encryptionkey@0"
I20260925 05:12:15.134125 29887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "91a79a129fba4d749d9ca45bcb0ac986"
format_stamp: "Formatted at 2026-09-25 05:12:15 on dist-test-slave-wpdm"
server_key: "29b4fb8f2c83ac7f2967b5b6b587ba06"
server_key_iv: "88c6c111f52688c6b541d4031c30fbd9"
server_key_version: "encryptionkey@0"
I20260925 05:12:15.139634 29887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 05:12:15.143535 29902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:15.145053 29887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:12:15.145258 29887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "91a79a129fba4d749d9ca45bcb0ac986"
format_stamp: "Formatted at 2026-09-25 05:12:15 on dist-test-slave-wpdm"
server_key: "29b4fb8f2c83ac7f2967b5b6b587ba06"
server_key_iv: "88c6c111f52688c6b541d4031c30fbd9"
server_key_version: "encryptionkey@0"
I20260925 05:12:15.145483 29887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:15.171082 29887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:15.175411 29887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:15.175750 29887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:15.195932 29887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39171
I20260925 05:12:15.195951 29953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39171 every 8 connection(s)
I20260925 05:12:15.197496 29887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:15.198716 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29887
I20260925 05:12:15.198922 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:15.199445 9052 external_mini_cluster.cc:1428] Setting key 039ed1a506a98655034d9f9c9fad902c
I20260925 05:12:15.202164 29954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:15.213827 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:15.201366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48955 (local address 127.0.0.1:39171)
0925 05:12:15.202231 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:15.202250 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:15.202311 (+ 61us) server_negotiation.cc:407] Connection header received
0925 05:12:15.204012 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:15.204036 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:15.204451 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:15.204935 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:15.205467 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:15.206844 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:15.208866 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:15.209585 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:15.209735 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:15.211150 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:15.211159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:15.211175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:15.211827 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:15.211917 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:15.211948 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:15.212061 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:15.212303 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 05:12:15.212687 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":394,"threads_started":1}
I20260925 05:12:15.214591 29954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986: Bootstrap starting.
I20260925 05:12:15.217839 29954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:15.217859 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:15.219049 29954 log.cc:824] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:15.222546 29954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986: No bootstrap required, opened a new log
I20260925 05:12:15.229737 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:15.223176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50255 (local address 127.0.0.1:39171)
0925 05:12:15.223368 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:15.223374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:15.223631 (+ 257us) server_negotiation.cc:407] Connection header received
0925 05:12:15.223797 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:15.223802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:15.223879 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:15.223991 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:15.224504 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:15.225678 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:15.227037 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:15.227753 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:15.227884 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:15.228948 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:15.228954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:15.228958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:15.228981 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:15.229032 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:15.229053 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:15.229160 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:15.229275 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 05:12:15.229377 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:12:15.232208 29954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a79a129fba4d749d9ca45bcb0ac986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260925 05:12:15.232594 29954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:15.232729 29954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a79a129fba4d749d9ca45bcb0ac986, State: Initialized, Role: FOLLOWER
I20260925 05:12:15.233433 29954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a79a129fba4d749d9ca45bcb0ac986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260925 05:12:15.233664 29954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:15.233840 29954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:15.234021 29954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:15.236249 29954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a79a129fba4d749d9ca45bcb0ac986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260925 05:12:15.236806 29954 leader_election.cc:302] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a79a129fba4d749d9ca45bcb0ac986; no voters:
I20260925 05:12:15.237391 29954 leader_election.cc:288] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:15.237524 29964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:15.238212 29964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [term 1 LEADER]: Becoming Leader. State: Replica: 91a79a129fba4d749d9ca45bcb0ac986, State: Running, Role: LEADER
I20260925 05:12:15.238757 29964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a79a129fba4d749d9ca45bcb0ac986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260925 05:12:15.239487 29954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:15.241905 29966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a79a129fba4d749d9ca45bcb0ac986. Latest consensus state: current_term: 1 leader_uuid: "91a79a129fba4d749d9ca45bcb0ac986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a79a129fba4d749d9ca45bcb0ac986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } } }
I20260925 05:12:15.242080 29965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a79a129fba4d749d9ca45bcb0ac986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a79a129fba4d749d9ca45bcb0ac986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } } }
I20260925 05:12:15.242404 29965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:15.242408 29966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:15.243468 29973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:15.249322 29973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:15.258783 29973 catalog_manager.cc:1347] Generated new cluster ID: 8205c8df527540a48555d1b5bc30c4d2
I20260925 05:12:15.258913 29973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:15.277838 29973 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:15.279196 29973 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:15.289677 29973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91a79a129fba4d749d9ca45bcb0ac986: Generated new TSK 0
I20260925 05:12:15.290572 29973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:15.299427 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29887
W20260925 05:12:15.300977 29949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48955
W20260925 05:12:15.301669 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:15.301827 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29887
2026-09-25T05:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:15 dist-test-slave-wpdm krb5kdc[29985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:15 dist-test-slave-wpdm krb5kdc[29985](info): set up 2 sockets
Sep 25 05:12:15 dist-test-slave-wpdm krb5kdc[29985](info): commencing operation
krb5kdc: starting...
W20260925 05:12:16.409623 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:16 dist-test-slave-wpdm krb5kdc[29985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:16.695369 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:17.030715 29998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:17.031008 29998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:17.031100 29998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:17.038434 29998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:17.038550 29998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:17.038601 29998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:17.038663 29998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:17.038707 29998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:17.048344 29998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:17.049608 29998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:17.050846 29998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:17.062613 30004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:17.062693 30006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:17.062711 30003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:17.063493 29998 server_base.cc:1034] running on GCE node
I20260925 05:12:17.064239 29998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:17.065747 29998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:17.066947 29998 hybrid_clock.cc:648] HybridClock initialized: now 1790313137066886 us; error 52 us; skew 500 ppm
I20260925 05:12:17.067272 29998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:17 dist-test-slave-wpdm krb5kdc[29985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:17.070426 29998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:17.071817 29998 webserver.cc:460] Webserver started at http://127.0.0.1:33569/ using document root <none> and password file <none>
I20260925 05:12:17.072443 29998 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:17.072543 29998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:17.072876 29998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:17.075737 29998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "dc97cf716fd94232a2eef59c538534ba"
format_stamp: "Formatted at 2026-09-25 05:12:17 on dist-test-slave-wpdm"
server_key: "a3bf37fd363dc08b700da9b3f472f718"
server_key_iv: "c5e28c292ff2e1457d5c6ef49f31404b"
server_key_version: "encryptionkey@0"
I20260925 05:12:17.076412 29998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "dc97cf716fd94232a2eef59c538534ba"
format_stamp: "Formatted at 2026-09-25 05:12:17 on dist-test-slave-wpdm"
server_key: "a3bf37fd363dc08b700da9b3f472f718"
server_key_iv: "c5e28c292ff2e1457d5c6ef49f31404b"
server_key_version: "encryptionkey@0"
I20260925 05:12:17.081513 29998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 05:12:17.084959 30013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:17.086282 29998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:12:17.086422 29998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "dc97cf716fd94232a2eef59c538534ba"
format_stamp: "Formatted at 2026-09-25 05:12:17 on dist-test-slave-wpdm"
server_key: "a3bf37fd363dc08b700da9b3f472f718"
server_key_iv: "c5e28c292ff2e1457d5c6ef49f31404b"
server_key_version: "encryptionkey@0"
I20260925 05:12:17.086598 29998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:17.092458 29998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:17.096526 29998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:17.096820 29998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:17.115372 29998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39993
I20260925 05:12:17.115412 30064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39993 every 8 connection(s)
I20260925 05:12:17.116852 29998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:17.120533 30065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:17.120549 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 29998
I20260925 05:12:17.120739 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:17.121234 9052 external_mini_cluster.cc:1428] Setting key 89951dd71c17eaa15a278399de58dd32
I20260925 05:12:17.128877 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:17.123008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48763 (local address 127.0.0.1:39993)
0925 05:12:17.123963 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:17.123975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:17.124014 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:12:17.125561 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:17.125582 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:17.125909 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:17.126239 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:17.127312 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:17.127317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:17.127328 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:17.127348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:17.127403 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:17.127421 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:17.127613 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:17.127710 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 05:12:17.127923 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":415,"threads_started":1}
I20260925 05:12:17.132561 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:17.134286 30065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba: Bootstrap starting.
I20260925 05:12:17.137630 30065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:17.139060 30065 log.cc:824] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:17.139411 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:17.136801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54601 (local address 127.0.0.1:39993)
0925 05:12:17.136987 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:17.136995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:17.137527 (+ 532us) server_negotiation.cc:407] Connection header received
0925 05:12:17.137725 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:17.137729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:17.137800 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:17.137877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:17.138810 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:17.138814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:17.138817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:17.138832 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:17.138869 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:17.138882 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:17.138969 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:17.139117 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 05:12:17.139161 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:12:17.141984 30065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba: No bootstrap required, opened a new log
I20260925 05:12:17.150029 30065 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc97cf716fd94232a2eef59c538534ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:12:17.150429 30065 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:17.150547 30065 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc97cf716fd94232a2eef59c538534ba, State: Initialized, Role: FOLLOWER
I20260925 05:12:17.151115 30065 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc97cf716fd94232a2eef59c538534ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:12:17.151355 30065 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:17.151508 30065 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:17.151688 30065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:17.154176 30065 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc97cf716fd94232a2eef59c538534ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:12:17.154671 30065 leader_election.cc:302] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc97cf716fd94232a2eef59c538534ba; no voters:
I20260925 05:12:17.155277 30065 leader_election.cc:288] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:17.155427 30075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:17.156221 30075 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [term 1 LEADER]: Becoming Leader. State: Replica: dc97cf716fd94232a2eef59c538534ba, State: Running, Role: LEADER
I20260925 05:12:17.156714 30075 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc97cf716fd94232a2eef59c538534ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:12:17.157521 30065 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:17.159533 30076 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc97cf716fd94232a2eef59c538534ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc97cf716fd94232a2eef59c538534ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260925 05:12:17.159852 30076 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:17.161703 30077 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc97cf716fd94232a2eef59c538534ba. Latest consensus state: current_term: 1 leader_uuid: "dc97cf716fd94232a2eef59c538534ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc97cf716fd94232a2eef59c538534ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260925 05:12:17.162009 30077 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:17.162233 30081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:17.166658 30081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:17.178754 30081 catalog_manager.cc:1347] Generated new cluster ID: 70b7afb8f5d740918183d24f734a64b3
I20260925 05:12:17.178866 30081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:17.197824 30081 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:17.198331 30081 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:17.215440 30081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc97cf716fd94232a2eef59c538534ba: Generated new TSK 0
I20260925 05:12:17.216567 30081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:17.231256 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29998
W20260925 05:12:17.232429 30060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48763
W20260925 05:12:17.233147 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 29998: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:17.233283 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 29998
2026-09-25T05:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:17 dist-test-slave-wpdm krb5kdc[30096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:17 dist-test-slave-wpdm krb5kdc[30096](info): set up 2 sockets
Sep 25 05:12:17 dist-test-slave-wpdm krb5kdc[30096](info): commencing operation
krb5kdc: starting...
W20260925 05:12:18.341583 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:18 dist-test-slave-wpdm krb5kdc[30096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:18.614137 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:18.963914 30109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:18.964228 30109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:18.964397 30109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:18.971644 30109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:18.971773 30109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:18.971872 30109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:18.971971 30109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:18.972051 30109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:18.983338 30109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:18.984639 30109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:18.985956 30109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:18.997963 30114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:18.998219 30115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:18.998240 30117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:18.998919 30109 server_base.cc:1034] running on GCE node
I20260925 05:12:18.999742 30109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:19.001286 30109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:19.002524 30109 hybrid_clock.cc:648] HybridClock initialized: now 1790313139002475 us; error 41 us; skew 500 ppm
I20260925 05:12:19.002874 30109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:19 dist-test-slave-wpdm krb5kdc[30096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:19.006100 30109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:19.007642 30109 webserver.cc:460] Webserver started at http://127.0.0.1:40637/ using document root <none> and password file <none>
I20260925 05:12:19.008337 30109 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:19.008481 30109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:19.008862 30109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:19.011790 30109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "3abd38b44aaf4839afe8a6daa035900f"
format_stamp: "Formatted at 2026-09-25 05:12:19 on dist-test-slave-wpdm"
server_key: "d67763569f0b3de5e4714de6b9abce80"
server_key_iv: "4fee3942a835a18d4d3160ea0c8b008f"
server_key_version: "encryptionkey@0"
I20260925 05:12:19.012555 30109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "3abd38b44aaf4839afe8a6daa035900f"
format_stamp: "Formatted at 2026-09-25 05:12:19 on dist-test-slave-wpdm"
server_key: "d67763569f0b3de5e4714de6b9abce80"
server_key_iv: "4fee3942a835a18d4d3160ea0c8b008f"
server_key_version: "encryptionkey@0"
I20260925 05:12:19.017807 30109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 05:12:19.021593 30124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:19.022887 30109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:12:19.023109 30109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "3abd38b44aaf4839afe8a6daa035900f"
format_stamp: "Formatted at 2026-09-25 05:12:19 on dist-test-slave-wpdm"
server_key: "d67763569f0b3de5e4714de6b9abce80"
server_key_iv: "4fee3942a835a18d4d3160ea0c8b008f"
server_key_version: "encryptionkey@0"
I20260925 05:12:19.023312 30109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:19.039772 30109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:19.043774 30109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:19.044099 30109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:19.063145 30109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35597
I20260925 05:12:19.063179 30175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35597 every 8 connection(s)
I20260925 05:12:19.064503 30109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:19.068177 30176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:19.070459 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30109
I20260925 05:12:19.070672 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:19.071223 9052 external_mini_cluster.cc:1428] Setting key fc5d497cb52117cfce5b67cc9381e4aa
I20260925 05:12:19.080646 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f: Bootstrap starting.
I20260925 05:12:19.084486 30176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:19.085753 30176 log.cc:824] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:12:19 dist-test-slave-wpdm krb5kdc[30096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:12:19.088985 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f: No bootstrap required, opened a new log
I20260925 05:12:19.094440 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:19.073173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:39827 (local address 127.0.0.1:35597)
0925 05:12:19.073981 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:19.073992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:19.074035 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:12:19.075663 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:19.075686 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:19.076061 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:19.076360 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:19.078147 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:19.079445 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:19.081272 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:19.081915 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:19.082045 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:19.086925 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:19.086939 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:19.086962 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:19.087011 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:19.090114 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:19.091049 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:19.091055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:19.091061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:19.091230 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:19.091701 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:19.091704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:19.091706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:19.092223 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:19.092457 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:19.093020 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 05:12:19.093382 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":391,"threads_started":1}
I20260925 05:12:19.098554 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:19.098353 30176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abd38b44aaf4839afe8a6daa035900f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:12:19.098803 30176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:19.098894 30176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abd38b44aaf4839afe8a6daa035900f, State: Initialized, Role: FOLLOWER
I20260925 05:12:19.099500 30176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abd38b44aaf4839afe8a6daa035900f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:12:19.099777 30176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:19.099895 30176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:19.100050 30176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:19.103363 30176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abd38b44aaf4839afe8a6daa035900f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:12:19.103964 30176 leader_election.cc:302] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3abd38b44aaf4839afe8a6daa035900f; no voters:
I20260925 05:12:19.104650 30176 leader_election.cc:288] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:19.104815 30186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:19.107420 30176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:19.109027 30186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [term 1 LEADER]: Becoming Leader. State: Replica: 3abd38b44aaf4839afe8a6daa035900f, State: Running, Role: LEADER
I20260925 05:12:19.109962 30186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abd38b44aaf4839afe8a6daa035900f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:12:19.113683 30187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3abd38b44aaf4839afe8a6daa035900f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abd38b44aaf4839afe8a6daa035900f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } } }
I20260925 05:12:19.114095 30187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:19.117318 30188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3abd38b44aaf4839afe8a6daa035900f. Latest consensus state: current_term: 1 leader_uuid: "3abd38b44aaf4839afe8a6daa035900f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abd38b44aaf4839afe8a6daa035900f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } } }
I20260925 05:12:19.117761 30188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:19.120151 30196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:19.121868 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:19.103586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:55849 (local address 127.0.0.1:35597)
0925 05:12:19.104018 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:19.104024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:19.104318 (+ 294us) server_negotiation.cc:407] Connection header received
0925 05:12:19.104513 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:19.104519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:19.104603 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:19.104723 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:19.106498 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:19.107374 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:19.108636 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:19.109406 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:19.109549 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:19.113054 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:19.113063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:19.113070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:19.113114 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:19.117782 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:19.119046 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:19.119058 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:19.119064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:19.119227 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:19.119888 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:19.119894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:19.119896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:19.120400 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:19.120659 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:19.121293 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 05:12:19.121505 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260925 05:12:19.125929 30196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:19.134701 30196 catalog_manager.cc:1347] Generated new cluster ID: 623c9c2a5a6645749c1a85326fb50eb8
I20260925 05:12:19.134812 30196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:19.143865 30196 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:19.145645 30196 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:19.162456 30196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3abd38b44aaf4839afe8a6daa035900f: Generated new TSK 0
I20260925 05:12:19.163350 30196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:19.171074 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30109
I20260925 05:12:19.172367 30171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:19.266280 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30109
W20260925 05:12:19.284066 30094 connection.cc:535] client connection to 127.0.0.1:35597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35597): Connection reset by peer (error 104)
2026-09-25T05:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:19 dist-test-slave-wpdm krb5kdc[30208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:19 dist-test-slave-wpdm krb5kdc[30208](info): set up 2 sockets
Sep 25 05:12:19 dist-test-slave-wpdm krb5kdc[30208](info): commencing operation
krb5kdc: starting...
W20260925 05:12:20.393568 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:20 dist-test-slave-wpdm krb5kdc[30208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:20.664986 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:20.999419 30221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:20.999704 30221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:20.999900 30221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:21.007391 30221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:21.007493 30221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:21.007616 30221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:21.007717 30221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:21.007798 30221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:21.018290 30221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:21.019656 30221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:21.021027 30221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:21.033182 30229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:21.033260 30226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:21.033281 30227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:21.034117 30221 server_base.cc:1034] running on GCE node
I20260925 05:12:21.035120 30221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:21.036597 30221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:21.037832 30221 hybrid_clock.cc:648] HybridClock initialized: now 1790313141037786 us; error 55 us; skew 500 ppm
I20260925 05:12:21.038199 30221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:21 dist-test-slave-wpdm krb5kdc[30208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:21.044513 30221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:21.046105 30221 webserver.cc:460] Webserver started at http://127.0.0.1:36041/ using document root <none> and password file <none>
I20260925 05:12:21.046802 30221 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:21.046950 30221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:21.047256 30221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:21.050324 30221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "e21af5a2f3e74e1eb2c54963b1e57192"
format_stamp: "Formatted at 2026-09-25 05:12:21 on dist-test-slave-wpdm"
server_key: "1e7dc814491381fbe653ed1c000dcefb"
server_key_iv: "0f81d447e7b9c234b51ba2e75330cc3f"
server_key_version: "encryptionkey@0"
I20260925 05:12:21.051156 30221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "e21af5a2f3e74e1eb2c54963b1e57192"
format_stamp: "Formatted at 2026-09-25 05:12:21 on dist-test-slave-wpdm"
server_key: "1e7dc814491381fbe653ed1c000dcefb"
server_key_iv: "0f81d447e7b9c234b51ba2e75330cc3f"
server_key_version: "encryptionkey@0"
I20260925 05:12:21.056289 30221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 05:12:21.059810 30236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:21.061182 30221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:21.061362 30221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "e21af5a2f3e74e1eb2c54963b1e57192"
format_stamp: "Formatted at 2026-09-25 05:12:21 on dist-test-slave-wpdm"
server_key: "1e7dc814491381fbe653ed1c000dcefb"
server_key_iv: "0f81d447e7b9c234b51ba2e75330cc3f"
server_key_version: "encryptionkey@0"
I20260925 05:12:21.061558 30221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:21.095786 30221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:21.100250 30221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:21.100579 30221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:21.120222 30221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35301
I20260925 05:12:21.120240 30287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35301 every 8 connection(s)
I20260925 05:12:21.121824 30221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:21.125689 30288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:21.131152 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30221
I20260925 05:12:21.131346 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:21.131883 9052 external_mini_cluster.cc:1428] Setting key 3457e23e6339abd1cc79c7362a27e4d1
I20260925 05:12:21.136814 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192: Bootstrap starting.
I20260925 05:12:21.140403 30288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:21.141656 30288 log.cc:824] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:21.145007 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192: No bootstrap required, opened a new log
I20260925 05:12:21.145589 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:21.133789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:36917 (local address 127.0.0.1:35301)
0925 05:12:21.134645 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:21.134656 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:21.134698 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:12:21.136304 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:21.136327 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:21.136803 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:21.137197 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:21.137785 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:21.139092 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:21.140937 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:21.141623 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:21.141869 (+ 246us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:21.143178 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:21.143186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:21.143200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:21.143709 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:21.143814 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:21.143847 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:21.144054 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:21.144390 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 05:12:21.144566 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":424,"threads_started":1}
I20260925 05:12:21.149511 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:21.153286 30288 raft_consensus.cc:348] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35301 } }
I20260925 05:12:21.153823 30288 raft_consensus.cc:374] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:21.153954 30288 raft_consensus.cc:729] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21af5a2f3e74e1eb2c54963b1e57192, State: Initialized, Role: FOLLOWER
I20260925 05:12:21.154613 30288 consensus_queue.cc:260] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35301 } }
I20260925 05:12:21.154877 30288 raft_consensus.cc:388] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:21.155032 30288 raft_consensus.cc:482] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:21.155215 30288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:21.157565 30288 raft_consensus.cc:504] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35301 } }
I20260925 05:12:21.158089 30288 leader_election.cc:302] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e21af5a2f3e74e1eb2c54963b1e57192; no voters:
I20260925 05:12:21.158778 30288 leader_election.cc:288] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:21.158869 30298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:21.161584 30298 raft_consensus.cc:686] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [term 1 LEADER]: Becoming Leader. State: Replica: e21af5a2f3e74e1eb2c54963b1e57192, State: Running, Role: LEADER
I20260925 05:12:21.161777 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:21.153778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:38555 (local address 127.0.0.1:35301)
0925 05:12:21.153963 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:21.153968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:21.154541 (+ 573us) server_negotiation.cc:407] Connection header received
0925 05:12:21.154749 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:21.154758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:21.154847 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:21.154942 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:21.155546 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:21.156361 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:21.157801 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:21.158318 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:21.158454 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:21.159672 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:21.159680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:21.159687 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:21.159712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:21.159785 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:21.159815 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:21.159930 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:21.160254 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 05:12:21.160325 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:12:21.162207 30298 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35301 } }
I20260925 05:12:21.162483 30288 sys_catalog.cc:564] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:21.165023 30299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35301 } } }
I20260925 05:12:21.165421 30299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:21.165915 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21af5a2f3e74e1eb2c54963b1e57192. Latest consensus state: current_term: 1 leader_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21af5a2f3e74e1eb2c54963b1e57192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35301 } } }
I20260925 05:12:21.166241 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:21.169898 30305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:21.175629 30305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:21.184114 30305 catalog_manager.cc:1347] Generated new cluster ID: 01dc9a7219a54b3091d22df6e50342af
I20260925 05:12:21.184228 30305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:21.208911 30305 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:21.210242 30305 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:21.217936 30305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e21af5a2f3e74e1eb2c54963b1e57192: Generated new TSK 0
I20260925 05:12:21.218786 30305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:21.224774 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30221
W20260925 05:12:21.226468 30283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.137:36917
W20260925 05:12:21.227252 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30221: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:21.227375 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30221
W20260925 05:12:21.245847 30206 connection.cc:535] client connection to 127.0.0.1:35301 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35301): Connection reset by peer (error 104)
2026-09-25T05:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:21 dist-test-slave-wpdm krb5kdc[30320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:21 dist-test-slave-wpdm krb5kdc[30320](info): set up 2 sockets
Sep 25 05:12:21 dist-test-slave-wpdm krb5kdc[30320](info): commencing operation
krb5kdc: starting...
W20260925 05:12:22.342046 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:22 dist-test-slave-wpdm krb5kdc[30320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:22.619269 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:22.959334 30333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:22.959657 30333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:22.959808 30333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:22.967581 30333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:22.967757 30333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:22.967867 30333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:22.967980 30333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:22.968065 30333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:22.978329 30333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:22.979682 30333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:22.981098 30333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:22.993126 30338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:22.993184 30339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:22.993113 30341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:22.994135 30333 server_base.cc:1034] running on GCE node
I20260925 05:12:22.994828 30333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:22.996531 30333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:22.997821 30333 hybrid_clock.cc:648] HybridClock initialized: now 1790313142997733 us; error 56 us; skew 500 ppm
I20260925 05:12:22.998302 30333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:23 dist-test-slave-wpdm krb5kdc[30320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:23.001524 30333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:23.002907 30333 webserver.cc:460] Webserver started at http://127.0.0.1:46235/ using document root <none> and password file <none>
I20260925 05:12:23.003609 30333 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:23.003762 30333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:23.004076 30333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:23.007009 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "8f88c2e79e374908acb4d4cdad4b02eb"
format_stamp: "Formatted at 2026-09-25 05:12:23 on dist-test-slave-wpdm"
server_key: "dea59e70f5f56a0fb07248a3411a5565"
server_key_iv: "5a325dfa29e0e7cfef687e1468587d59"
server_key_version: "encryptionkey@0"
I20260925 05:12:23.007762 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "8f88c2e79e374908acb4d4cdad4b02eb"
format_stamp: "Formatted at 2026-09-25 05:12:23 on dist-test-slave-wpdm"
server_key: "dea59e70f5f56a0fb07248a3411a5565"
server_key_iv: "5a325dfa29e0e7cfef687e1468587d59"
server_key_version: "encryptionkey@0"
I20260925 05:12:23.012688 30333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:12:23.016111 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:23.017500 30333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:23.017686 30333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "8f88c2e79e374908acb4d4cdad4b02eb"
format_stamp: "Formatted at 2026-09-25 05:12:23 on dist-test-slave-wpdm"
server_key: "dea59e70f5f56a0fb07248a3411a5565"
server_key_iv: "5a325dfa29e0e7cfef687e1468587d59"
server_key_version: "encryptionkey@0"
I20260925 05:12:23.017887 30333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:23.023407 30333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:23.027771 30333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:23.028079 30333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:23.046231 30333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38921
I20260925 05:12:23.046276 30399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38921 every 8 connection(s)
I20260925 05:12:23.047647 30333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:23.051245 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:23.055269 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30333
I20260925 05:12:23.055507 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:23.056002 9052 external_mini_cluster.cc:1428] Setting key f48fb45adfdf40259a5862896b307f4f
I20260925 05:12:23.063522 30400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb: Bootstrap starting.
I20260925 05:12:23.065557 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:23.057841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:43713 (local address 127.0.0.1:38921)
0925 05:12:23.058610 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:23.058625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:23.058677 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:12:23.061697 (+ 3020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:23.061725 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:23.062151 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:23.062583 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:23.063582 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:23.063602 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:23.063619 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:23.063638 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:23.063712 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:23.063738 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:23.063964 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:23.064115 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 05:12:23.064303 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":305,"threads_started":1}
I20260925 05:12:23.067142 30400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:23.068361 30400 log.cc:824] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:23.069202 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:23.071501 30400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb: No bootstrap required, opened a new log
I20260925 05:12:23.076644 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:23.074361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:51241 (local address 127.0.0.1:38921)
0925 05:12:23.074574 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:23.074579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:23.074850 (+ 271us) server_negotiation.cc:407] Connection header received
0925 05:12:23.075005 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:23.075012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:23.075059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:23.075135 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:23.076014 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:23.076018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:23.076021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:23.076038 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:23.076072 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:23.076088 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:23.076175 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:23.076308 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 05:12:23.076354 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:12:23.080489 30400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260925 05:12:23.080943 30400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:23.081076 30400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f88c2e79e374908acb4d4cdad4b02eb, State: Initialized, Role: FOLLOWER
I20260925 05:12:23.081729 30400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260925 05:12:23.082005 30400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:23.082156 30400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:23.082321 30400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:23.084447 30400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260925 05:12:23.084996 30400 leader_election.cc:302] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f88c2e79e374908acb4d4cdad4b02eb; no voters:
I20260925 05:12:23.085553 30400 leader_election.cc:288] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:23.085757 30410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:23.086586 30410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [term 1 LEADER]: Becoming Leader. State: Replica: 8f88c2e79e374908acb4d4cdad4b02eb, State: Running, Role: LEADER
I20260925 05:12:23.087255 30410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260925 05:12:23.087823 30400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:23.090659 30411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } } }
I20260925 05:12:23.091079 30411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:23.091696 30412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f88c2e79e374908acb4d4cdad4b02eb. Latest consensus state: current_term: 1 leader_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f88c2e79e374908acb4d4cdad4b02eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } } }
I20260925 05:12:23.092074 30412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:23.092279 30418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:23.098443 30418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:23.108139 30418 catalog_manager.cc:1347] Generated new cluster ID: 3eb0b9a57d14467a939547eaf7b5524a
I20260925 05:12:23.108304 30418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:23.132792 30418 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:23.133327 30418 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:23.158008 30418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f88c2e79e374908acb4d4cdad4b02eb: Generated new TSK 0
I20260925 05:12:23.159166 30418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:23.169310 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30333
W20260925 05:12:23.170603 30395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.137:43713
W20260925 05:12:23.171437 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:23.171607 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30333
2026-09-25T05:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:23 dist-test-slave-wpdm krb5kdc[30431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:23 dist-test-slave-wpdm krb5kdc[30431](info): set up 2 sockets
Sep 25 05:12:23 dist-test-slave-wpdm krb5kdc[30431](info): commencing operation
krb5kdc: starting...
W20260925 05:12:24.313898 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:24 dist-test-slave-wpdm krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:24.581512 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:24.909924 30444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:24.910190 30444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:24.910368 30444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:24.917712 30444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:24.917842 30444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:24.917924 30444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:24.918027 30444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:24.918088 30444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:24.927994 30444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:24.929451 30444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:24.930822 30444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:24.945436 30449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:24.945983 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:24.946096 30450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:24.946875 30444 server_base.cc:1034] running on GCE node
I20260925 05:12:24.947624 30444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:24.949029 30444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:24.950251 30444 hybrid_clock.cc:648] HybridClock initialized: now 1790313144950149 us; error 97 us; skew 500 ppm
I20260925 05:12:24.950776 30444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:24 dist-test-slave-wpdm krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:24.954352 30444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:24.955734 30444 webserver.cc:460] Webserver started at http://127.0.0.1:38321/ using document root <none> and password file <none>
I20260925 05:12:24.956378 30444 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:24.956475 30444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:24.956768 30444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:24.959728 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "d787eb453ae14eb3a73637f38c98385a"
format_stamp: "Formatted at 2026-09-25 05:12:24 on dist-test-slave-wpdm"
server_key: "76da01afdcea5299f9419be1122f0865"
server_key_iv: "bd9f98c71994076bebbb70fc97ecdf45"
server_key_version: "encryptionkey@0"
I20260925 05:12:24.960417 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "d787eb453ae14eb3a73637f38c98385a"
format_stamp: "Formatted at 2026-09-25 05:12:24 on dist-test-slave-wpdm"
server_key: "76da01afdcea5299f9419be1122f0865"
server_key_iv: "bd9f98c71994076bebbb70fc97ecdf45"
server_key_version: "encryptionkey@0"
I20260925 05:12:24.965705 30444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:12:24.969098 30459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:24.970350 30444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:12:24.970528 30444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "d787eb453ae14eb3a73637f38c98385a"
format_stamp: "Formatted at 2026-09-25 05:12:24 on dist-test-slave-wpdm"
server_key: "76da01afdcea5299f9419be1122f0865"
server_key_iv: "bd9f98c71994076bebbb70fc97ecdf45"
server_key_version: "encryptionkey@0"
I20260925 05:12:24.970707 30444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:24.992144 30444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:24.997140 30444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:24.997424 30444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:25.021515 30444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34165
I20260925 05:12:25.021543 30510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34165 every 8 connection(s)
I20260925 05:12:25.022954 30444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:25.026626 30511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:25.028489 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30444
I20260925 05:12:25.028676 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:25.029264 9052 external_mini_cluster.cc:1428] Setting key 5cf02b85f6c078b3d36bb1cb3805224f
I20260925 05:12:25.039398 30511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a: Bootstrap starting.
I20260925 05:12:25.042510 30511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a: Neither blocks nor log segments found. Creating new log.
Sep 25 05:12:25 dist-test-slave-wpdm krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:12:25.043778 30511 log.cc:824] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:25.046722 30511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a: No bootstrap required, opened a new log
I20260925 05:12:25.051543 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:25.031148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:51231 (local address 127.0.0.1:34165)
0925 05:12:25.031892 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:25.031904 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:25.031945 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:12:25.033641 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:25.033666 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:25.034062 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:25.034451 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:25.035937 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:25.037325 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:25.038566 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:25.039234 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:25.039439 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:25.044008 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:25.044017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:25.044031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:25.044066 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:25.047412 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:25.048301 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:25.048307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:25.048313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:25.048481 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:25.048968 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:25.048971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:25.048973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:25.049469 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:25.049706 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:25.050207 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 05:12:25.050505 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":319,"threads_started":1}
I20260925 05:12:25.055616 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:25.056078 30511 raft_consensus.cc:348] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d787eb453ae14eb3a73637f38c98385a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34165 } }
I20260925 05:12:25.056478 30511 raft_consensus.cc:374] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:25.056582 30511 raft_consensus.cc:729] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d787eb453ae14eb3a73637f38c98385a, State: Initialized, Role: FOLLOWER
I20260925 05:12:25.057247 30511 consensus_queue.cc:260] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d787eb453ae14eb3a73637f38c98385a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34165 } }
I20260925 05:12:25.057523 30511 raft_consensus.cc:388] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:25.057644 30511 raft_consensus.cc:482] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:25.057797 30511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:25.061021 30511 raft_consensus.cc:504] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d787eb453ae14eb3a73637f38c98385a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34165 } }
I20260925 05:12:25.061633 30511 leader_election.cc:302] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d787eb453ae14eb3a73637f38c98385a; no voters:
I20260925 05:12:25.062422 30511 leader_election.cc:288] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:25.062637 30521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:25.063256 30521 raft_consensus.cc:686] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [term 1 LEADER]: Becoming Leader. State: Replica: d787eb453ae14eb3a73637f38c98385a, State: Running, Role: LEADER
I20260925 05:12:25.063885 30521 consensus_queue.cc:237] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d787eb453ae14eb3a73637f38c98385a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34165 } }
I20260925 05:12:25.070819 30511 sys_catalog.cc:564] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:25.071389 30523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d787eb453ae14eb3a73637f38c98385a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d787eb453ae14eb3a73637f38c98385a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34165 } } }
I20260925 05:12:25.071874 30523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:25.080829 30522 sys_catalog.cc:455] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d787eb453ae14eb3a73637f38c98385a. Latest consensus state: current_term: 1 leader_uuid: "d787eb453ae14eb3a73637f38c98385a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d787eb453ae14eb3a73637f38c98385a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34165 } } }
I20260925 05:12:25.081143 30522 sys_catalog.cc:458] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:25.081925 30526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:25.081947 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:25.060653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:40665 (local address 127.0.0.1:34165)
0925 05:12:25.060896 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:25.060906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:25.061291 (+ 385us) server_negotiation.cc:407] Connection header received
0925 05:12:25.061502 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:25.061507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:25.061595 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:25.061709 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:25.064108 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:25.065252 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:25.067543 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:25.068501 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:25.068680 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:25.072464 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:25.072471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:25.072475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:25.072510 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:25.076871 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:25.078018 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:25.078027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:25.078033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:25.078220 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:25.078907 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:25.078914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:25.078920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:25.079599 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:25.079900 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:25.080894 (+ 994us) server_negotiation.cc:299] Negotiation successful
0925 05:12:25.081229 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 05:12:25.092393 30526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:25.110443 30526 catalog_manager.cc:1347] Generated new cluster ID: ab3ae535ec134e83882b71b749afe0b8
I20260925 05:12:25.110613 30526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:25.131701 30526 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:25.133883 30526 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:25.142690 30526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d787eb453ae14eb3a73637f38c98385a: Generated new TSK 0
I20260925 05:12:25.143510 30526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:25.150805 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30444
I20260925 05:12:25.151880 30506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:25.245817 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30444
W20260925 05:12:25.264042 30429 connection.cc:535] client connection to 127.0.0.1:34165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34165): Connection reset by peer (error 104)
2026-09-25T05:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:25 dist-test-slave-wpdm krb5kdc[30544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:25 dist-test-slave-wpdm krb5kdc[30544](info): set up 2 sockets
Sep 25 05:12:25 dist-test-slave-wpdm krb5kdc[30544](info): commencing operation
krb5kdc: starting...
W20260925 05:12:26.365650 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:26 dist-test-slave-wpdm krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:26.660398 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:27.004585 30557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:27.005013 30557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:27.005127 30557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:27.013233 30557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:27.013415 30557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:27.013499 30557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:27.013612 30557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:27.013679 30557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:27.024065 30557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:27.025488 30557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:27.026811 30557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:27.038933 30562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:27.039628 30565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:27.039672 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:27.040611 30557 server_base.cc:1034] running on GCE node
I20260925 05:12:27.041604 30557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:27.043215 30557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:27.044493 30557 hybrid_clock.cc:648] HybridClock initialized: now 1790313147044442 us; error 68 us; skew 500 ppm
I20260925 05:12:27.045051 30557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:27 dist-test-slave-wpdm krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:27.048594 30557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:27.050392 30557 webserver.cc:460] Webserver started at http://127.0.0.1:37673/ using document root <none> and password file <none>
I20260925 05:12:27.051154 30557 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:27.051327 30557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:27.051676 30557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:27.055112 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "2865598fc03a47a39dc2638a9a74d372"
format_stamp: "Formatted at 2026-09-25 05:12:27 on dist-test-slave-wpdm"
server_key: "5edcc6f3cc43b50d8aaa7b1d63201857"
server_key_iv: "5746d0f0ef79912f059b56cbbdc9a039"
server_key_version: "encryptionkey@0"
I20260925 05:12:27.056028 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "2865598fc03a47a39dc2638a9a74d372"
format_stamp: "Formatted at 2026-09-25 05:12:27 on dist-test-slave-wpdm"
server_key: "5edcc6f3cc43b50d8aaa7b1d63201857"
server_key_iv: "5746d0f0ef79912f059b56cbbdc9a039"
server_key_version: "encryptionkey@0"
I20260925 05:12:27.062233 30557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 05:12:27.066323 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:27.067844 30557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:12:27.068207 30557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "2865598fc03a47a39dc2638a9a74d372"
format_stamp: "Formatted at 2026-09-25 05:12:27 on dist-test-slave-wpdm"
server_key: "5edcc6f3cc43b50d8aaa7b1d63201857"
server_key_iv: "5746d0f0ef79912f059b56cbbdc9a039"
server_key_version: "encryptionkey@0"
I20260925 05:12:27.068464 30557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:27.091109 30557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:27.095715 30557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:27.096055 30557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:27.115783 30557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35141
I20260925 05:12:27.115806 30623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35141 every 8 connection(s)
I20260925 05:12:27.117435 30557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:27.121446 30624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:27.126107 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30557
I20260925 05:12:27.126325 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:27.126850 9052 external_mini_cluster.cc:1428] Setting key 74f6ecd9e6699f27a0805137490a327d
I20260925 05:12:27.133872 30624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372: Bootstrap starting.
I20260925 05:12:27.137796 30624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:27.139003 30624 log.cc:824] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:27.142781 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:27.129072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:35147 (local address 127.0.0.1:35141)
0925 05:12:27.130080 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:27.130093 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:27.130141 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:12:27.132274 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:27.132300 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:27.132875 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:27.133316 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:27.133979 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:27.135493 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:27.137344 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:27.138198 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:27.138429 (+ 231us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:27.139918 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:27.139927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:27.139943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:27.140616 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:27.140737 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:27.140773 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:27.140947 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:27.141235 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 05:12:27.141413 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":411,"threads_started":1}
I20260925 05:12:27.142827 30624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372: No bootstrap required, opened a new log
I20260925 05:12:27.147182 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:27.153139 30624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865598fc03a47a39dc2638a9a74d372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35141 } }
I20260925 05:12:27.153759 30624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:27.153934 30624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2865598fc03a47a39dc2638a9a74d372, State: Initialized, Role: FOLLOWER
I20260925 05:12:27.154676 30624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865598fc03a47a39dc2638a9a74d372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35141 } }
I20260925 05:12:27.155019 30624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:27.155195 30624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:27.155385 30624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:27.158795 30624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865598fc03a47a39dc2638a9a74d372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35141 } }
I20260925 05:12:27.159497 30624 leader_election.cc:302] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2865598fc03a47a39dc2638a9a74d372; no voters:
I20260925 05:12:27.159674 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:27.151774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:41947 (local address 127.0.0.1:35141)
0925 05:12:27.152026 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:27.152033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:27.152518 (+ 485us) server_negotiation.cc:407] Connection header received
0925 05:12:27.152815 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:27.152819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:27.152935 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:27.153035 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:27.153797 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:27.154687 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:27.156782 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:27.157303 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:27.157403 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:27.158726 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:27.158731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:27.158734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:27.158753 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:27.158823 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:27.158839 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:27.158950 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:27.159208 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 05:12:27.159255 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:12:27.160380 30624 leader_election.cc:288] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:27.160537 30634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:27.163405 30624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:27.163849 30634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [term 1 LEADER]: Becoming Leader. State: Replica: 2865598fc03a47a39dc2638a9a74d372, State: Running, Role: LEADER
I20260925 05:12:27.164541 30634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865598fc03a47a39dc2638a9a74d372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35141 } }
I20260925 05:12:27.167851 30636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2865598fc03a47a39dc2638a9a74d372. Latest consensus state: current_term: 1 leader_uuid: "2865598fc03a47a39dc2638a9a74d372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865598fc03a47a39dc2638a9a74d372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35141 } } }
I20260925 05:12:27.168272 30636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:27.168844 30635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2865598fc03a47a39dc2638a9a74d372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865598fc03a47a39dc2638a9a74d372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35141 } } }
I20260925 05:12:27.169154 30635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:27.173228 30644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:27.178776 30644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:27.188653 30644 catalog_manager.cc:1347] Generated new cluster ID: 9df17a32c1ab42c9872c827dc426a05c
I20260925 05:12:27.188825 30644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:27.203883 30644 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:27.205282 30644 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:27.212218 30644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2865598fc03a47a39dc2638a9a74d372: Generated new TSK 0
I20260925 05:12:27.213207 30644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:27.223315 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30557
W20260925 05:12:27.224429 30619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.137:35147
W20260925 05:12:27.225257 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30557: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:27.225414 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30557
W20260925 05:12:27.244174 30542 connection.cc:535] client connection to 127.0.0.1:35141 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35141): Connection reset by peer (error 104)
2026-09-25T05:12:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:27 dist-test-slave-wpdm krb5kdc[30655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:27 dist-test-slave-wpdm krb5kdc[30655](info): set up 2 sockets
Sep 25 05:12:27 dist-test-slave-wpdm krb5kdc[30655](info): commencing operation
krb5kdc: starting...
W20260925 05:12:28.425753 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:28 dist-test-slave-wpdm krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:12:28.696547 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:29.030725 30668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:29.031044 30668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:29.031239 30668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:29.038785 30668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:29.038906 30668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:29.039037 30668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:29.039162 30668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:29.039232 30668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:29.049427 30668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:29.050877 30668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:29.052222 30668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:29.064350 30674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:29.064320 30673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:29.064401 30676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:29.065281 30668 server_base.cc:1034] running on GCE node
I20260925 05:12:29.066191 30668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:29.067613 30668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:29.068835 30668 hybrid_clock.cc:648] HybridClock initialized: now 1790313149068778 us; error 54 us; skew 500 ppm
I20260925 05:12:29.069186 30668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:29 dist-test-slave-wpdm krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:29.072322 30668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:12:29.073948 30668 webserver.cc:460] Webserver started at http://127.0.0.1:44415/ using document root <none> and password file <none>
I20260925 05:12:29.074652 30668 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:29.074800 30668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:29.075117 30668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:29.078223 30668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "951a5eb05e574fba93c3e29d0ced350e"
format_stamp: "Formatted at 2026-09-25 05:12:29 on dist-test-slave-wpdm"
server_key: "8fa4fe5668d3aca4dfe1784adf511c5b"
server_key_iv: "1d74647109097c6fd2478c942304f0aa"
server_key_version: "encryptionkey@0"
I20260925 05:12:29.079022 30668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "951a5eb05e574fba93c3e29d0ced350e"
format_stamp: "Formatted at 2026-09-25 05:12:29 on dist-test-slave-wpdm"
server_key: "8fa4fe5668d3aca4dfe1784adf511c5b"
server_key_iv: "1d74647109097c6fd2478c942304f0aa"
server_key_version: "encryptionkey@0"
I20260925 05:12:29.084439 30668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:12:29.088100 30683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:29.089509 30668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:29.089701 30668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "951a5eb05e574fba93c3e29d0ced350e"
format_stamp: "Formatted at 2026-09-25 05:12:29 on dist-test-slave-wpdm"
server_key: "8fa4fe5668d3aca4dfe1784adf511c5b"
server_key_iv: "1d74647109097c6fd2478c942304f0aa"
server_key_version: "encryptionkey@0"
I20260925 05:12:29.089905 30668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:29.096117 30668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:29.099967 30668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:29.100288 30668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:29.119414 30668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36419
I20260925 05:12:29.119504 30734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36419 every 8 connection(s)
I20260925 05:12:29.120985 30668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:29.121934 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30668
I20260925 05:12:29.122117 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:29.122514 9052 external_mini_cluster.cc:1428] Setting key a58ed47c42f9868ef5cb5260f57b3671
I20260925 05:12:29.125480 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:29.130067 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:29.124246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:49947 (local address 127.0.0.1:36419)
0925 05:12:29.125057 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:29.125068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:29.125109 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:12:29.126725 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:29.126746 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:29.127084 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:29.127400 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:29.128463 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:29.128469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:29.128481 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:29.128504 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:29.128577 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:29.128596 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:29.128830 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:29.128929 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:12:29.129153 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":352,"threads_started":1}
I20260925 05:12:29.133808 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:29.138406 30735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e: Bootstrap starting.
I20260925 05:12:29.141695 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:29.139210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:34719 (local address 127.0.0.1:36419)
0925 05:12:29.139417 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:29.139423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:29.139739 (+ 316us) server_negotiation.cc:407] Connection header received
0925 05:12:29.139909 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:29.139912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:29.139959 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:29.140037 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:29.141041 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:29.141044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:29.141048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:29.141065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:29.141102 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:29.141116 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:29.141202 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:29.141337 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 05:12:29.141385 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 05:12:29.142434 30735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:29.143533 30735 log.cc:824] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:29.146585 30735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e: No bootstrap required, opened a new log
I20260925 05:12:29.154325 30735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951a5eb05e574fba93c3e29d0ced350e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260925 05:12:29.154762 30735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:29.154881 30735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951a5eb05e574fba93c3e29d0ced350e, State: Initialized, Role: FOLLOWER
I20260925 05:12:29.155463 30735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951a5eb05e574fba93c3e29d0ced350e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260925 05:12:29.155701 30735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:29.155848 30735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:29.156049 30735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:29.158455 30735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951a5eb05e574fba93c3e29d0ced350e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260925 05:12:29.158921 30735 leader_election.cc:302] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951a5eb05e574fba93c3e29d0ced350e; no voters:
I20260925 05:12:29.159456 30735 leader_election.cc:288] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:29.159607 30745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:29.160286 30745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [term 1 LEADER]: Becoming Leader. State: Replica: 951a5eb05e574fba93c3e29d0ced350e, State: Running, Role: LEADER
I20260925 05:12:29.160873 30745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951a5eb05e574fba93c3e29d0ced350e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260925 05:12:29.161614 30735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:29.164008 30747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951a5eb05e574fba93c3e29d0ced350e. Latest consensus state: current_term: 1 leader_uuid: "951a5eb05e574fba93c3e29d0ced350e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951a5eb05e574fba93c3e29d0ced350e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } } }
I20260925 05:12:29.164409 30747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:29.164520 30746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951a5eb05e574fba93c3e29d0ced350e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951a5eb05e574fba93c3e29d0ced350e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } } }
I20260925 05:12:29.164919 30746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:29.167806 30754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:29.173049 30754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:29.182018 30754 catalog_manager.cc:1347] Generated new cluster ID: 7c220028dc11476e8791b1936edbfc85
I20260925 05:12:29.182185 30754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:29.193259 30754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:29.193812 30754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:29.204236 30754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 951a5eb05e574fba93c3e29d0ced350e: Generated new TSK 0
I20260925 05:12:29.205171 30754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:29.212100 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30668
W20260925 05:12:29.213253 30730 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.137:49947
W20260925 05:12:29.213963 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:29.214114 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30668
2026-09-25T05:12:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:29 dist-test-slave-wpdm krb5kdc[30766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:29 dist-test-slave-wpdm krb5kdc[30766](info): set up 2 sockets
Sep 25 05:12:29 dist-test-slave-wpdm krb5kdc[30766](info): commencing operation
krb5kdc: starting...
W20260925 05:12:30.357645 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:30 dist-test-slave-wpdm krb5kdc[30766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:30.632798 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:45399
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:45399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:30.971982 30779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:30.972334 30779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:30.972503 30779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:30.980120 30779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:30.980283 30779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:30.980391 30779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:30.980499 30779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:30.980579 30779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:30.990780 30779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:45399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:45399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:30.992146 30779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:30.993728 30779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:31.005779 30787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:31.005888 30784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:31.005915 30785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:31.006811 30779 server_base.cc:1034] running on GCE node
I20260925 05:12:31.007596 30779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:31.009083 30779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:31.010306 30779 hybrid_clock.cc:648] HybridClock initialized: now 1790313151010261 us; error 58 us; skew 500 ppm
I20260925 05:12:31.010691 30779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:31 dist-test-slave-wpdm krb5kdc[30766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313151, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:31.013806 30779 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:31.015295 30779 webserver.cc:460] Webserver started at http://127.8.215.62:41371/ using document root <none> and password file <none>
I20260925 05:12:31.015978 30779 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:31.016127 30779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:31.016481 30779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:31.019482 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b"
format_stamp: "Formatted at 2026-09-25 05:12:31 on dist-test-slave-wpdm"
server_key: "e5ddec7fa0fee59c71a9f3694ed78df1"
server_key_iv: "74dff4c6dde5fdf0b062dea971e02df6"
server_key_version: "encryptionkey@0"
I20260925 05:12:31.020256 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b"
format_stamp: "Formatted at 2026-09-25 05:12:31 on dist-test-slave-wpdm"
server_key: "e5ddec7fa0fee59c71a9f3694ed78df1"
server_key_iv: "74dff4c6dde5fdf0b062dea971e02df6"
server_key_version: "encryptionkey@0"
I20260925 05:12:31.025583 30779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:12:31.030035 30794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:31.031819 30779 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260925 05:12:31.031998 30779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b"
format_stamp: "Formatted at 2026-09-25 05:12:31 on dist-test-slave-wpdm"
server_key: "e5ddec7fa0fee59c71a9f3694ed78df1"
server_key_iv: "74dff4c6dde5fdf0b062dea971e02df6"
server_key_version: "encryptionkey@0"
I20260925 05:12:31.032194 30779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:31.062194 30779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:31.066759 30779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:31.067062 30779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:31.086158 30779 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:45399
I20260925 05:12:31.086187 30845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:45399 every 8 connection(s)
I20260925 05:12:31.087635 30779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:31.089838 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30779
I20260925 05:12:31.090052 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:31.090495 9052 external_mini_cluster.cc:1428] Setting key cff7c6558ad4cfb65b83d94364fda7db
I20260925 05:12:31.091912 30846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:31.105367 30846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b: Bootstrap starting.
Sep 25 05:12:31 dist-test-slave-wpdm krb5kdc[30766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:31.108569 30846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:31.109942 30846 log.cc:824] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:31.113197 30846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b: No bootstrap required, opened a new log
I20260925 05:12:31.117712 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:31.092293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47003 (local address 127.8.215.62:45399)
0925 05:12:31.093105 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:31.093116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:31.097276 (+ 4160us) server_negotiation.cc:407] Connection header received
0925 05:12:31.099363 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:31.099385 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:31.099787 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:31.100172 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:31.101642 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:31.102936 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:31.104243 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:31.105053 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:31.105248 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:31.109859 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:31.109867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:31.109881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:31.109916 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:31.113364 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:31.114353 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:31.114360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:31.114365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:31.114525 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:31.115022 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:31.115025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:31.115027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:31.115557 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:31.115797 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:31.116295 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 05:12:31.116624 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":381,"threads_started":1}
I20260925 05:12:31.121523 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:31.122169 30846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45399 } }
I20260925 05:12:31.122517 30846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:31.122608 30846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b70b11b3ed44d5e9d97334bcf0cb41b, State: Initialized, Role: FOLLOWER
I20260925 05:12:31.123176 30846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45399 } }
I20260925 05:12:31.123397 30846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:31.123553 30846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:31.123709 30846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:31.126457 30846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45399 } }
I20260925 05:12:31.127044 30846 leader_election.cc:302] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b70b11b3ed44d5e9d97334bcf0cb41b; no voters:
I20260925 05:12:31.127709 30846 leader_election.cc:288] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:31.127784 30856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:31.129917 30846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:31.134619 30856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [term 1 LEADER]: Becoming Leader. State: Replica: 2b70b11b3ed44d5e9d97334bcf0cb41b, State: Running, Role: LEADER
I20260925 05:12:31.135331 30856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45399 } }
I20260925 05:12:31.142932 30857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45399 } } }
I20260925 05:12:31.143507 30857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:31.144906 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b70b11b3ed44d5e9d97334bcf0cb41b. Latest consensus state: current_term: 1 leader_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b70b11b3ed44d5e9d97334bcf0cb41b" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 45399 } } }
I20260925 05:12:31.145349 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:31.146992 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:31.126163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37635 (local address 127.8.215.62:45399)
0925 05:12:31.126379 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:31.126388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:31.126772 (+ 384us) server_negotiation.cc:407] Connection header received
0925 05:12:31.126952 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:31.126956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:31.127013 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:31.127102 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:31.129171 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:31.130796 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:31.133314 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:31.134525 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:31.134683 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:31.138751 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:31.138761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:31.138769 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:31.138827 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:31.142953 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:31.144217 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:31.144225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:31.144231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:31.144409 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:31.145055 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:31.145059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:31.145062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:31.145624 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:31.145952 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:31.146301 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 05:12:31.146571 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:12:31.147523 30871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:31.152968 30871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:31.161921 30871 catalog_manager.cc:1347] Generated new cluster ID: 1e0cc5267e0b428ab462e769040656ca
I20260925 05:12:31.162026 30871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:31.177676 30871 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:31.178956 30871 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:31.195792 30871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b70b11b3ed44d5e9d97334bcf0cb41b: Generated new TSK 0
I20260925 05:12:31.196657 30871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:31.203521 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30779
I20260925 05:12:31.204655 30841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:31.295608 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30779
W20260925 05:12:31.313488 30764 connection.cc:535] client connection to 127.8.215.62:45399 recv error: Network error: failed to read from TLS socket (remote: 127.8.215.62:45399): Connection reset by peer (error 104)
2026-09-25T05:12:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:31 dist-test-slave-wpdm krb5kdc[30878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:31 dist-test-slave-wpdm krb5kdc[30878](info): set up 2 sockets
Sep 25 05:12:31 dist-test-slave-wpdm krb5kdc[30878](info): commencing operation
krb5kdc: starting...
W20260925 05:12:32.409675 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:32 dist-test-slave-wpdm krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:32.692855 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:39451
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:39451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:33.035872 30891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:33.036274 30891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:33.036480 30891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:33.045257 30891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:33.045418 30891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:33.045508 30891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:33.045598 30891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:33.045686 30891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:33.055860 30891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:39451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:39451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:33.057260 30891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:33.058574 30891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:33.070384 30896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:33.070451 30897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:33.070493 30899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:33.071774 30891 server_base.cc:1034] running on GCE node
I20260925 05:12:33.072535 30891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:33.074151 30891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:33.075371 30891 hybrid_clock.cc:648] HybridClock initialized: now 1790313153075328 us; error 42 us; skew 500 ppm
I20260925 05:12:33.075734 30891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:33 dist-test-slave-wpdm krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313153, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:33.082754 30891 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:33.084298 30891 webserver.cc:460] Webserver started at http://127.8.215.62:37653/ using document root <none> and password file <none>
I20260925 05:12:33.085099 30891 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:33.085232 30891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:33.085544 30891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:33.088563 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "1512162f9eff45ceba1c9e0f95477a1f"
format_stamp: "Formatted at 2026-09-25 05:12:33 on dist-test-slave-wpdm"
server_key: "3233b68111d5ab8e1f968d6daecd9630"
server_key_iv: "a770b6c8cbc109d0b89867d54d956511"
server_key_version: "encryptionkey@0"
I20260925 05:12:33.089423 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "1512162f9eff45ceba1c9e0f95477a1f"
format_stamp: "Formatted at 2026-09-25 05:12:33 on dist-test-slave-wpdm"
server_key: "3233b68111d5ab8e1f968d6daecd9630"
server_key_iv: "a770b6c8cbc109d0b89867d54d956511"
server_key_version: "encryptionkey@0"
I20260925 05:12:33.094553 30891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:12:33.098031 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:33.099336 30891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 05:12:33.099511 30891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "1512162f9eff45ceba1c9e0f95477a1f"
format_stamp: "Formatted at 2026-09-25 05:12:33 on dist-test-slave-wpdm"
server_key: "3233b68111d5ab8e1f968d6daecd9630"
server_key_iv: "a770b6c8cbc109d0b89867d54d956511"
server_key_version: "encryptionkey@0"
I20260925 05:12:33.099706 30891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:33.117794 30891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:33.121942 30891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:33.122249 30891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:33.141453 30891 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:39451
I20260925 05:12:33.141511 30957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:39451 every 8 connection(s)
I20260925 05:12:33.142885 30891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:33.146611 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:33.151280 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 30891
I20260925 05:12:33.151461 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:33.151998 9052 external_mini_cluster.cc:1428] Setting key 18199cab3bff81a435bca74784e7bc1a
I20260925 05:12:33.158005 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f: Bootstrap starting.
I20260925 05:12:33.161099 30958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:33.162164 30958 log.cc:824] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:33.165542 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f: No bootstrap required, opened a new log
I20260925 05:12:33.165668 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:33.153944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39415 (local address 127.8.215.62:39451)
0925 05:12:33.154741 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:33.154756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:33.154797 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:12:33.156681 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:33.156703 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:33.157149 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:33.157514 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:33.158092 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:33.159388 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:33.160720 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:33.161491 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:33.161703 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:33.163045 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:33.163052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:33.163065 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:33.163686 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:33.163780 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:33.163810 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:33.163982 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:33.164229 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:12:33.164398 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":373,"threads_started":1}
I20260925 05:12:33.169430 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:33.173730 30958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512162f9eff45ceba1c9e0f95477a1f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39451 } }
I20260925 05:12:33.174240 30958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:33.174360 30958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1512162f9eff45ceba1c9e0f95477a1f, State: Initialized, Role: FOLLOWER
I20260925 05:12:33.175040 30958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512162f9eff45ceba1c9e0f95477a1f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39451 } }
I20260925 05:12:33.175339 30958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:33.175539 30958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:33.175737 30958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:33.178895 30958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512162f9eff45ceba1c9e0f95477a1f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39451 } }
I20260925 05:12:33.179497 30958 leader_election.cc:302] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1512162f9eff45ceba1c9e0f95477a1f; no voters:
I20260925 05:12:33.180217 30958 leader_election.cc:288] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:33.180380 30968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:33.181069 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:33.173788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57773 (local address 127.8.215.62:39451)
0925 05:12:33.174034 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:33.174044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:33.174440 (+ 396us) server_negotiation.cc:407] Connection header received
0925 05:12:33.174640 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:33.174644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:33.174750 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:33.174857 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:33.175405 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:33.176358 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:33.178249 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:33.178709 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:33.178800 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:33.180034 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:33.180043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:33.180050 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:33.180077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:33.180131 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:33.180159 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:33.180320 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:33.180561 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 05:12:33.180626 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:12:33.182382 30968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [term 1 LEADER]: Becoming Leader. State: Replica: 1512162f9eff45ceba1c9e0f95477a1f, State: Running, Role: LEADER
I20260925 05:12:33.183000 30958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:33.183082 30968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512162f9eff45ceba1c9e0f95477a1f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39451 } }
I20260925 05:12:33.186081 30970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1512162f9eff45ceba1c9e0f95477a1f. Latest consensus state: current_term: 1 leader_uuid: "1512162f9eff45ceba1c9e0f95477a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512162f9eff45ceba1c9e0f95477a1f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39451 } } }
I20260925 05:12:33.186115 30969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1512162f9eff45ceba1c9e0f95477a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512162f9eff45ceba1c9e0f95477a1f" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 39451 } } }
I20260925 05:12:33.186486 30969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:33.186486 30970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:33.187354 30976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:33.193025 30976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:33.202323 30976 catalog_manager.cc:1347] Generated new cluster ID: 28404e6a7b604ad8b5604ad1f7b4e654
I20260925 05:12:33.202442 30976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:33.247432 30976 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:33.249358 30976 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:33.261607 30976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1512162f9eff45ceba1c9e0f95477a1f: Generated new TSK 0
I20260925 05:12:33.262666 30976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:33.274869 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30891
W20260925 05:12:33.276029 30953 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39415
W20260925 05:12:33.276822 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 30891: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:33.276968 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 30891
W20260925 05:12:33.292117 30876 connection.cc:535] client connection to 127.8.215.62:39451 recv error: Network error: failed to read from TLS socket (remote: 127.8.215.62:39451): Connection reset by peer (error 104)
2026-09-25T05:12:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:33 dist-test-slave-wpdm krb5kdc[30989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:33 dist-test-slave-wpdm krb5kdc[30989](info): set up 2 sockets
Sep 25 05:12:33 dist-test-slave-wpdm krb5kdc[30989](info): commencing operation
krb5kdc: starting...
W20260925 05:12:34.401651 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:34 dist-test-slave-wpdm krb5kdc[30989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:34.676338 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35143
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:35143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:35.017740 31002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:35.018110 31002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:35.018282 31002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:35.025738 31002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:35.025928 31002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:35.026036 31002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:35.026145 31002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:35.026232 31002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:35.036345 31002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:35143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:35.037746 31002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:35.039104 31002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:35.052197 31007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:35.052301 31010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:35.052294 31008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:35.053328 31002 server_base.cc:1034] running on GCE node
I20260925 05:12:35.054270 31002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:35.055850 31002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:35.057083 31002 hybrid_clock.cc:648] HybridClock initialized: now 1790313155057046 us; error 45 us; skew 500 ppm
I20260925 05:12:35.057440 31002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:35 dist-test-slave-wpdm krb5kdc[30989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313155, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:35.060889 31002 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:35.062361 31002 webserver.cc:460] Webserver started at http://127.8.215.62:42497/ using document root <none> and password file <none>
I20260925 05:12:35.063048 31002 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:35.063148 31002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:35.063403 31002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:35.066498 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "147dc2a98e724686a6da37d8911b3e51"
format_stamp: "Formatted at 2026-09-25 05:12:35 on dist-test-slave-wpdm"
server_key: "643082f8a7d017862552c50f5c251302"
server_key_iv: "3caba84e476189620864af4d6719cddb"
server_key_version: "encryptionkey@0"
I20260925 05:12:35.067241 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "147dc2a98e724686a6da37d8911b3e51"
format_stamp: "Formatted at 2026-09-25 05:12:35 on dist-test-slave-wpdm"
server_key: "643082f8a7d017862552c50f5c251302"
server_key_iv: "3caba84e476189620864af4d6719cddb"
server_key_version: "encryptionkey@0"
I20260925 05:12:35.072464 31002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:12:35.076059 31017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:35.077484 31002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:35.077692 31002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "147dc2a98e724686a6da37d8911b3e51"
format_stamp: "Formatted at 2026-09-25 05:12:35 on dist-test-slave-wpdm"
server_key: "643082f8a7d017862552c50f5c251302"
server_key_iv: "3caba84e476189620864af4d6719cddb"
server_key_version: "encryptionkey@0"
I20260925 05:12:35.077898 31002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:35.083699 31002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:35.088227 31002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:35.088541 31002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:35.108062 31002 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:35143
I20260925 05:12:35.108108 31068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:35143 every 8 connection(s)
I20260925 05:12:35.109648 31002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:35.113780 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:35.114197 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31002
I20260925 05:12:35.114384 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:35.114817 9052 external_mini_cluster.cc:1428] Setting key 4e1aa8d28dfa3dac0f78ef25760f3928
I20260925 05:12:35.123102 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:35.116625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58173 (local address 127.8.215.62:35143)
0925 05:12:35.117574 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:35.117591 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:35.117642 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:12:35.119203 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:35.119238 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:35.119700 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:35.120155 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:35.121160 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:35.121169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:35.121190 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:35.121207 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:35.121271 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:35.121295 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:35.121536 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:35.121661 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 05:12:35.121858 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":375,"threads_started":1}
I20260925 05:12:35.126780 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:35.128217 31069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51: Bootstrap starting.
I20260925 05:12:35.131644 31069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:35.133070 31069 log.cc:824] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:35.133850 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:35.131150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36407 (local address 127.8.215.62:35143)
0925 05:12:35.131381 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:35.131389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:35.131759 (+ 370us) server_negotiation.cc:407] Connection header received
0925 05:12:35.131930 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:35.131934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:35.131984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:35.132073 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:35.133099 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:35.133102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:35.133106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:35.133120 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:35.133158 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:35.133194 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:35.133331 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:35.133462 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 05:12:35.133522 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:12:35.136371 31069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51: No bootstrap required, opened a new log
I20260925 05:12:35.143771 31069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dc2a98e724686a6da37d8911b3e51" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35143 } }
I20260925 05:12:35.144140 31069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:35.144260 31069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147dc2a98e724686a6da37d8911b3e51, State: Initialized, Role: FOLLOWER
I20260925 05:12:35.144939 31069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dc2a98e724686a6da37d8911b3e51" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35143 } }
I20260925 05:12:35.145208 31069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:35.145359 31069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:35.145537 31069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:35.147686 31069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dc2a98e724686a6da37d8911b3e51" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35143 } }
I20260925 05:12:35.148187 31069 leader_election.cc:302] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147dc2a98e724686a6da37d8911b3e51; no voters:
I20260925 05:12:35.148869 31069 leader_election.cc:288] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:35.149075 31079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:35.150076 31079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [term 1 LEADER]: Becoming Leader. State: Replica: 147dc2a98e724686a6da37d8911b3e51, State: Running, Role: LEADER
I20260925 05:12:35.150606 31079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dc2a98e724686a6da37d8911b3e51" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35143 } }
I20260925 05:12:35.151166 31069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:35.153228 31081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147dc2a98e724686a6da37d8911b3e51. Latest consensus state: current_term: 1 leader_uuid: "147dc2a98e724686a6da37d8911b3e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dc2a98e724686a6da37d8911b3e51" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35143 } } }
I20260925 05:12:35.153652 31081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:35.154124 31080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147dc2a98e724686a6da37d8911b3e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dc2a98e724686a6da37d8911b3e51" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35143 } } }
I20260925 05:12:35.154405 31080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:35.157706 31086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:35.162868 31086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:35.172107 31086 catalog_manager.cc:1347] Generated new cluster ID: 5fdfdc5981194b9497baf16839ddd9a2
I20260925 05:12:35.172286 31086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:35.195564 31086 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:35.196167 31086 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:35.204409 31086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147dc2a98e724686a6da37d8911b3e51: Generated new TSK 0
I20260925 05:12:35.205279 31086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:35.211091 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31002
W20260925 05:12:35.212213 31064 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58173
W20260925 05:12:35.213115 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31002: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:35.213285 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31002
2026-09-25T05:12:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:35 dist-test-slave-wpdm krb5kdc[31101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:35 dist-test-slave-wpdm krb5kdc[31101](info): set up 2 sockets
Sep 25 05:12:35 dist-test-slave-wpdm krb5kdc[31101](info): commencing operation
krb5kdc: starting...
W20260925 05:12:36.325721 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:36 dist-test-slave-wpdm krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:36.627522 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35245
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:35245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:36.964628 31114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:36.965026 31114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:36.965198 31114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:36.972281 31114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:36.972445 31114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:36.972556 31114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:36.972664 31114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:36.972769 31114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:36.982889 31114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:35245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:35245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:36.984277 31114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:36.985682 31114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:36.998042 31122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:36.997989 31119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:36.998095 31120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:37.002326 31114 server_base.cc:1034] running on GCE node
I20260925 05:12:37.003122 31114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:37.004638 31114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:37.005869 31114 hybrid_clock.cc:648] HybridClock initialized: now 1790313157005814 us; error 47 us; skew 500 ppm
I20260925 05:12:37.006219 31114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:37 dist-test-slave-wpdm krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:37.009305 31114 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:37.010798 31114 webserver.cc:460] Webserver started at http://127.8.215.62:43275/ using document root <none> and password file <none>
I20260925 05:12:37.011487 31114 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:37.011612 31114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:37.011910 31114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:37.015064 31114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "ca04ffd595154a07ac26844d29de24c3"
format_stamp: "Formatted at 2026-09-25 05:12:37 on dist-test-slave-wpdm"
server_key: "e16e09a0f20fbd887d01f4aed1d4629f"
server_key_iv: "a443daa14895767fe4a738b4beced37c"
server_key_version: "encryptionkey@0"
I20260925 05:12:37.015820 31114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "ca04ffd595154a07ac26844d29de24c3"
format_stamp: "Formatted at 2026-09-25 05:12:37 on dist-test-slave-wpdm"
server_key: "e16e09a0f20fbd887d01f4aed1d4629f"
server_key_iv: "a443daa14895767fe4a738b4beced37c"
server_key_version: "encryptionkey@0"
I20260925 05:12:37.021044 31114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:12:37.024554 31129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:37.025873 31114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:37.026067 31114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "ca04ffd595154a07ac26844d29de24c3"
format_stamp: "Formatted at 2026-09-25 05:12:37 on dist-test-slave-wpdm"
server_key: "e16e09a0f20fbd887d01f4aed1d4629f"
server_key_iv: "a443daa14895767fe4a738b4beced37c"
server_key_version: "encryptionkey@0"
I20260925 05:12:37.026237 31114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:37.040505 31114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:37.044361 31114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:37.044647 31114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:37.063766 31114 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:35245
I20260925 05:12:37.063789 31180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:35245 every 8 connection(s)
I20260925 05:12:37.065303 31114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:37.069092 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:37.075109 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31114
I20260925 05:12:37.075306 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:37.075733 9052 external_mini_cluster.cc:1428] Setting key cb44238ad82597a2572bde84fbfe48b5
I20260925 05:12:37.079963 31181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3: Bootstrap starting.
I20260925 05:12:37.083101 31181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:37.084146 31181 log.cc:824] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:37.086953 31181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3: No bootstrap required, opened a new log
Sep 25 05:12:37 dist-test-slave-wpdm krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:37.096395 31181 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca04ffd595154a07ac26844d29de24c3" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35245 } }
I20260925 05:12:37.096971 31181 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:37.097078 31181 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca04ffd595154a07ac26844d29de24c3, State: Initialized, Role: FOLLOWER
I20260925 05:12:37.097669 31181 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca04ffd595154a07ac26844d29de24c3" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35245 } }
I20260925 05:12:37.097899 31181 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:37.098066 31181 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:37.098258 31181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:37.098860 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:37.077641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37963 (local address 127.8.215.62:35245)
0925 05:12:37.078508 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:37.078520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:37.078560 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:12:37.080315 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:37.080339 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:37.080764 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:37.081145 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:37.082591 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:37.083972 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:37.085387 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:37.086046 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:37.086233 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:37.090866 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:37.090873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:37.090892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:37.090931 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:37.094125 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:37.094977 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:37.094983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:37.094989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:37.095158 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:37.095547 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:37.095550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:37.095552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:37.096088 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:37.096296 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:37.096862 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 05:12:37.097598 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":400,"threads_started":1}
I20260925 05:12:37.101158 31181 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca04ffd595154a07ac26844d29de24c3" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35245 } }
I20260925 05:12:37.101776 31181 leader_election.cc:302] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca04ffd595154a07ac26844d29de24c3; no voters:
I20260925 05:12:37.102476 31181 leader_election.cc:288] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:37.102519 31186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:37.102984 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:37.103293 31186 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [term 1 LEADER]: Becoming Leader. State: Replica: ca04ffd595154a07ac26844d29de24c3, State: Running, Role: LEADER
I20260925 05:12:37.103803 31186 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca04ffd595154a07ac26844d29de24c3" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35245 } }
I20260925 05:12:37.106580 31181 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:37.108026 31188 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca04ffd595154a07ac26844d29de24c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca04ffd595154a07ac26844d29de24c3" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35245 } } }
I20260925 05:12:37.108598 31188 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:37.108110 31187 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca04ffd595154a07ac26844d29de24c3. Latest consensus state: current_term: 1 leader_uuid: "ca04ffd595154a07ac26844d29de24c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca04ffd595154a07ac26844d29de24c3" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 35245 } } }
I20260925 05:12:37.108950 31187 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:37.121865 31193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:37.127687 31193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:37.141305 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:37.118884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46703 (local address 127.8.215.62:35245)
0925 05:12:37.119127 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:37.119137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:37.119676 (+ 539us) server_negotiation.cc:407] Connection header received
0925 05:12:37.119895 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:37.119910 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:37.119980 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:37.120101 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:37.122084 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:37.123051 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:37.125524 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:37.126529 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:37.126695 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:37.130895 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:37.130905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:37.130913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:37.130967 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:37.136047 (+ 5080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:37.137271 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:37.137279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:37.137287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:37.137479 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:37.138039 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:37.138047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:37.138053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:37.138585 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:37.138887 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:37.140372 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0925 05:12:37.140716 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:12:37.147576 31193 catalog_manager.cc:1347] Generated new cluster ID: 76d46735719f4bef92b1ea7dfdfbb400
I20260925 05:12:37.147714 31193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:37.168167 31193 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:37.169836 31193 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:37.181592 31193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca04ffd595154a07ac26844d29de24c3: Generated new TSK 0
I20260925 05:12:37.182420 31193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:37.190285 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31114
I20260925 05:12:37.191525 31176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:37.283830 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31114
2026-09-25T05:12:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:37 dist-test-slave-wpdm krb5kdc[31213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:37 dist-test-slave-wpdm krb5kdc[31213](info): set up 2 sockets
Sep 25 05:12:37 dist-test-slave-wpdm krb5kdc[31213](info): commencing operation
krb5kdc: starting...
W20260925 05:12:38.389936 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:38 dist-test-slave-wpdm krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:38.675510 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:34897
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:34897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:39.016265 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:39.016670 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:39.016860 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:39.024583 31226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:39.024777 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:39.024873 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:39.024947 31226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:39.024991 31226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:39.035632 31226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:34897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:39.037000 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:39.038326 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:39.051043 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:39.050973 31234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:39.051016 31232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:39.052199 31226 server_base.cc:1034] running on GCE node
I20260925 05:12:39.053114 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:39.054567 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:39.055819 31226 hybrid_clock.cc:648] HybridClock initialized: now 1790313159055768 us; error 35 us; skew 500 ppm
I20260925 05:12:39.056197 31226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:39 dist-test-slave-wpdm krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313159, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:39.063477 31226 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:39.065449 31226 webserver.cc:460] Webserver started at http://127.8.215.62:39277/ using document root <none> and password file <none>
I20260925 05:12:39.066221 31226 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:39.066342 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:39.066736 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:39.070163 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "0da19c0551854d449e68f515396d873e"
format_stamp: "Formatted at 2026-09-25 05:12:39 on dist-test-slave-wpdm"
server_key: "38ce70409f3a21ef6193c7ee67261aca"
server_key_iv: "843d0e0947f98b2d78c645ea82dbc4d6"
server_key_version: "encryptionkey@0"
I20260925 05:12:39.071059 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "0da19c0551854d449e68f515396d873e"
format_stamp: "Formatted at 2026-09-25 05:12:39 on dist-test-slave-wpdm"
server_key: "38ce70409f3a21ef6193c7ee67261aca"
server_key_iv: "843d0e0947f98b2d78c645ea82dbc4d6"
server_key_version: "encryptionkey@0"
I20260925 05:12:39.077030 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 05:12:39.081364 31241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:39.082809 31226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 05:12:39.083019 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "0da19c0551854d449e68f515396d873e"
format_stamp: "Formatted at 2026-09-25 05:12:39 on dist-test-slave-wpdm"
server_key: "38ce70409f3a21ef6193c7ee67261aca"
server_key_iv: "843d0e0947f98b2d78c645ea82dbc4d6"
server_key_version: "encryptionkey@0"
I20260925 05:12:39.083232 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:39.103907 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:39.108178 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:39.108489 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:39.128471 31226 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:34897
I20260925 05:12:39.128525 31292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:34897 every 8 connection(s)
I20260925 05:12:39.129916 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:39.132447 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31226
I20260925 05:12:39.132654 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:39.133152 9052 external_mini_cluster.cc:1428] Setting key 12e45a6ab5100bc54bb9edc44d0c30e0
I20260925 05:12:39.134055 31293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:39.147771 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:39.135061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37401 (local address 127.8.215.62:34897)
0925 05:12:39.135842 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:39.135857 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:39.135908 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:12:39.137718 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:39.137746 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:39.138288 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:39.138751 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:39.139267 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:39.141320 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:39.142602 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:39.143639 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:39.143831 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:39.144968 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:39.144978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:39.145002 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:39.145761 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:39.145862 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:39.145898 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:39.146030 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:39.146200 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 05:12:39.146480 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":319,"threads_started":1}
I20260925 05:12:39.149117 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e: Bootstrap starting.
I20260925 05:12:39.151535 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:39.152318 31293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:39.153435 31293 log.cc:824] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:39.156669 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e: No bootstrap required, opened a new log
I20260925 05:12:39.163306 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:39.156454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57871 (local address 127.8.215.62:34897)
0925 05:12:39.156627 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:39.156634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:39.157102 (+ 468us) server_negotiation.cc:407] Connection header received
0925 05:12:39.157258 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:39.157263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:39.157340 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:39.157452 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:39.157943 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:39.159096 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:39.160520 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:39.161266 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:39.161395 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:39.162510 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:39.162517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:39.162521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:39.162542 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:39.162590 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:39.162609 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:39.162715 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:39.162871 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 05:12:39.162971 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:12:39.166013 31293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da19c0551854d449e68f515396d873e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34897 } }
I20260925 05:12:39.166358 31293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:39.166428 31293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da19c0551854d449e68f515396d873e, State: Initialized, Role: FOLLOWER
I20260925 05:12:39.166913 31293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da19c0551854d449e68f515396d873e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34897 } }
I20260925 05:12:39.167099 31293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:39.167191 31293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:39.167320 31293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:39.169711 31293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da19c0551854d449e68f515396d873e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34897 } }
I20260925 05:12:39.170213 31293 leader_election.cc:302] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da19c0551854d449e68f515396d873e; no voters:
I20260925 05:12:39.170809 31293 leader_election.cc:288] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:39.170939 31303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:39.171612 31303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [term 1 LEADER]: Becoming Leader. State: Replica: 0da19c0551854d449e68f515396d873e, State: Running, Role: LEADER
I20260925 05:12:39.172103 31303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da19c0551854d449e68f515396d873e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34897 } }
I20260925 05:12:39.172890 31293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:39.175369 31304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0da19c0551854d449e68f515396d873e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da19c0551854d449e68f515396d873e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34897 } } }
I20260925 05:12:39.175875 31304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:39.176465 31305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da19c0551854d449e68f515396d873e. Latest consensus state: current_term: 1 leader_uuid: "0da19c0551854d449e68f515396d873e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da19c0551854d449e68f515396d873e" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 34897 } } }
I20260925 05:12:39.176944 31305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:39.180873 31310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:39.185997 31310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:39.195474 31310 catalog_manager.cc:1347] Generated new cluster ID: 518f2b8b438f4cc7ab7a51a642bfbd3c
I20260925 05:12:39.195628 31310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:39.206967 31310 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:39.208227 31310 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:39.222208 31310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da19c0551854d449e68f515396d873e: Generated new TSK 0
I20260925 05:12:39.223140 31310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:39.240419 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31226
W20260925 05:12:39.241539 31288 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37401
W20260925 05:12:39.242274 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31226: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:39.242439 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31226
2026-09-25T05:12:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:39 dist-test-slave-wpdm krb5kdc[31324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:39 dist-test-slave-wpdm krb5kdc[31324](info): set up 2 sockets
Sep 25 05:12:39 dist-test-slave-wpdm krb5kdc[31324](info): commencing operation
krb5kdc: starting...
W20260925 05:12:40.365828 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:40 dist-test-slave-wpdm krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:40.645540 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43511
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:40.983497 31337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:40.983826 31337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:40.984030 31337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:40.991518 31337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:40.991645 31337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:40.991773 31337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:40.991900 31337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:40.991971 31337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:41.002332 31337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:41.003726 31337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:41.005172 31337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:41.018179 31343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:41.018247 31345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:41.018146 31342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:41.019677 31337 server_base.cc:1034] running on GCE node
I20260925 05:12:41.020424 31337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:41.022001 31337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:41.023200 31337 hybrid_clock.cc:648] HybridClock initialized: now 1790313161023141 us; error 51 us; skew 500 ppm
I20260925 05:12:41.023550 31337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:41 dist-test-slave-wpdm krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313161, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:41.026891 31337 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:41.028373 31337 webserver.cc:460] Webserver started at http://127.8.215.62:38717/ using document root <none> and password file <none>
I20260925 05:12:41.029147 31337 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:41.029292 31337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:41.029613 31337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:41.032655 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "c6e53c73677d4b3c8acb5ec2b908c134"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-wpdm"
server_key: "ee3f8011fcc3633cfecbdaa2d851f47d"
server_key_iv: "7d110cf3c822d6500580b52e9a916cc8"
server_key_version: "encryptionkey@0"
I20260925 05:12:41.033473 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "c6e53c73677d4b3c8acb5ec2b908c134"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-wpdm"
server_key: "ee3f8011fcc3633cfecbdaa2d851f47d"
server_key_iv: "7d110cf3c822d6500580b52e9a916cc8"
server_key_version: "encryptionkey@0"
I20260925 05:12:41.038527 31337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:12:41.042431 31352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:41.043658 31337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:12:41.043857 31337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "c6e53c73677d4b3c8acb5ec2b908c134"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-wpdm"
server_key: "ee3f8011fcc3633cfecbdaa2d851f47d"
server_key_iv: "7d110cf3c822d6500580b52e9a916cc8"
server_key_version: "encryptionkey@0"
I20260925 05:12:41.044050 31337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:41.049698 31337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:41.054076 31337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:41.054371 31337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:41.072904 31337 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:43511
I20260925 05:12:41.072940 31403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:43511 every 8 connection(s)
I20260925 05:12:41.074345 31337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:41.078115 31404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:41.082546 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31337
I20260925 05:12:41.082741 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:41.083184 9052 external_mini_cluster.cc:1428] Setting key c415aa3bd6e94916d4e1f088f27bde57
I20260925 05:12:41.090726 31404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134: Bootstrap starting.
I20260925 05:12:41.092594 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:41.085027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48863 (local address 127.8.215.62:43511)
0925 05:12:41.085866 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:41.085880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:41.085930 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:12:41.088797 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:41.088825 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:41.089271 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:41.089715 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:41.090682 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:41.090690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:41.090712 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:41.090731 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:41.090802 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:41.090833 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:41.091049 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:41.091171 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 05:12:41.091373 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":332,"threads_started":1}
I20260925 05:12:41.094525 31404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:41.095785 31404 log.cc:824] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:41.096427 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:41.098541 31404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134: No bootstrap required, opened a new log
I20260925 05:12:41.102984 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:41.100675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36015 (local address 127.8.215.62:43511)
0925 05:12:41.100855 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:41.100861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:41.101242 (+ 381us) server_negotiation.cc:407] Connection header received
0925 05:12:41.101364 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:41.101369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:41.101427 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:41.101525 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:41.102392 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:41.102398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:41.102402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:12:41.102425 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:41.102469 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:12:41.102493 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:41.102602 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:41.102671 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:12:41.102722 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:12:41.107190 31404 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43511 } }
I20260925 05:12:41.107520 31404 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:41.107647 31404 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e53c73677d4b3c8acb5ec2b908c134, State: Initialized, Role: FOLLOWER
I20260925 05:12:41.108139 31404 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43511 } }
I20260925 05:12:41.108343 31404 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:41.108439 31404 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:41.108566 31404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:41.111081 31404 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43511 } }
I20260925 05:12:41.111526 31404 leader_election.cc:302] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6e53c73677d4b3c8acb5ec2b908c134; no voters:
I20260925 05:12:41.112066 31404 leader_election.cc:288] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:41.112213 31414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:41.112914 31414 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [term 1 LEADER]: Becoming Leader. State: Replica: c6e53c73677d4b3c8acb5ec2b908c134, State: Running, Role: LEADER
I20260925 05:12:41.113448 31414 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43511 } }
I20260925 05:12:41.114207 31404 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:41.116320 31415 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43511 } } }
I20260925 05:12:41.117201 31415 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:41.116873 31416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6e53c73677d4b3c8acb5ec2b908c134. Latest consensus state: current_term: 1 leader_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e53c73677d4b3c8acb5ec2b908c134" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 43511 } } }
I20260925 05:12:41.117738 31416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:41.118932 31420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:41.123816 31420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:41.134354 31420 catalog_manager.cc:1347] Generated new cluster ID: a3113ad71ec2432a989dde5815133dde
I20260925 05:12:41.134526 31420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:41.154050 31420 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:41.154476 31420 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:41.162962 31420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6e53c73677d4b3c8acb5ec2b908c134: Generated new TSK 0
I20260925 05:12:41.163982 31420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:41.178064 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31337
W20260925 05:12:41.179075 31399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48863
W20260925 05:12:41.179765 9052 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31337: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:12:41.179894 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31337
2026-09-25T05:12:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1939 ms)
[----------] 18 tests from AuthTokenIssuingTest (35978 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:41 dist-test-slave-wpdm krb5kdc[31435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:41 dist-test-slave-wpdm krb5kdc[31435](info): set up 2 sockets
Sep 25 05:12:41 dist-test-slave-wpdm krb5kdc[31435](info): commencing operation
krb5kdc: starting...
W20260925 05:12:42.289934 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:42 dist-test-slave-wpdm krb5kdc[31435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:42.567937 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:36075
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36075,127.8.215.61:41533,127.8.215.60:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:42.911753 31448 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:12:42.912184 31448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:42.912333 31448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:42.912487 31448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:42.920120 31448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:42.920265 31448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:42.920377 31448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:42.920477 31448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:42.920619 31448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:42.930990 31448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36075,127.8.215.61:41533,127.8.215.60:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:42.932412 31448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:42.934176 31448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:42.949445 31454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.949656 31453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.950176 31456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:42.950724 31448 server_base.cc:1034] running on GCE node
I20260925 05:12:42.951474 31448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:42.952889 31448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:42.954118 31448 hybrid_clock.cc:648] HybridClock initialized: now 1790313162954048 us; error 40 us; skew 500 ppm
I20260925 05:12:42.954457 31448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:42 dist-test-slave-wpdm krb5kdc[31435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:42.957711 31448 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:42.959232 31448 webserver.cc:460] Webserver started at http://127.8.215.62:45633/ using document root <none> and password file <none>
I20260925 05:12:42.959937 31448 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:42.960072 31448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:42.960413 31448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:42.963452 31448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "cda77d14988845e79c49d35a332818bd"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-wpdm"
server_key: "a90fa4c4f43800b99fe1b181768836b9"
server_key_iv: "17c593d5edb47f09381e702617bd1d09"
server_key_version: "encryptionkey@0"
I20260925 05:12:42.964274 31448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "cda77d14988845e79c49d35a332818bd"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-wpdm"
server_key: "a90fa4c4f43800b99fe1b181768836b9"
server_key_iv: "17c593d5edb47f09381e702617bd1d09"
server_key_version: "encryptionkey@0"
I20260925 05:12:42.969600 31448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:12:42.973047 31463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.974431 31448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:12:42.974620 31448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "cda77d14988845e79c49d35a332818bd"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-wpdm"
server_key: "a90fa4c4f43800b99fe1b181768836b9"
server_key_iv: "17c593d5edb47f09381e702617bd1d09"
server_key_version: "encryptionkey@0"
I20260925 05:12:42.974853 31448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:42.989619 31448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:42.993470 31448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:42.993706 31448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:43.013875 31448 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:36075
I20260925 05:12:43.013895 31514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:36075 every 8 connection(s)
I20260925 05:12:43.015344 31448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:43.016095 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31448
I20260925 05:12:43.016348 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:43.017140 9052 external_mini_cluster.cc:1428] Setting key 83258eeede122a93b5cb9bab5ca21c93
I20260925 05:12:43.019637 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:43.026839 31515 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.215.61@KRBTEST.COM; defaulting to no policy
Sep 25 05:12:43 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.61@KRBTEST.COM" created.
I20260925 05:12:43.051617 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.030357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.8.215.62:36075)
0925 05:12:43.031527 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.031544 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.032152 (+ 608us) server_negotiation.cc:407] Connection header received
0925 05:12:43.034414 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.034424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.034774 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.034998 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.036255 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.038325 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.039912 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.041003 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.041206 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.044595 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.044603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.044616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.044643 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.047471 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.048370 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.048377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.048379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.048520 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.048993 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.048997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.048999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.049476 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.049624 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.050272 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.050566 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":877,"spinlock_wait_cycles":16640,"thread_start_us":619,"threads_started":1}
I20260925 05:12:43.051662 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.030357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36075 (local address 127.0.0.1:41684)
0925 05:12:43.031956 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.032010 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:43.034014 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:43.035036 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:43.035041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:43.035661 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:43.036032 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.036050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.038500 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.038504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.039733 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.039758 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.041241 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.041246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.041376 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.042047 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:43.042069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:43.044365 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:43.047610 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.047625 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.047641 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.048235 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.048646 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.048655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.048661 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.048887 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.049629 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:43.049639 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:43.049898 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 05:12:43.050197 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 05:12:43.050739 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"mutex_wait_us":49,"thread_start_us":1075,"threads_started":1}
I20260925 05:12:43.054477 31515 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:43.055722 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.055212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50782)
0925 05:12:43.055436 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.055572 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 05:12:43.056018 31465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
I20260925 05:12:43.057569 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.057244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50794)
0925 05:12:43.057376 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.057441 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:12:43.057814 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:41533: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab.
Entry for principal kudu/127.8.215.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab.
I20260925 05:12:43.080315 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:41533
--webserver_interface=127.8.215.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab
--principal=kudu/127.8.215.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36075,127.8.215.61:41533,127.8.215.60:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:43.107199 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } attempt: 1
I20260925 05:12:43.108353 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.107887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50796)
0925 05:12:43.108120 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.108211 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 05:12:43.109444 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.109099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50812)
0925 05:12:43.109264 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.109332 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:12:43.109710 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:41533: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
I20260925 05:12:43.177951 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } attempt: 2
I20260925 05:12:43.179260 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.178770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50826)
0925 05:12:43.179023 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.179126 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260925 05:12:43.180295 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.179974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50838)
0925 05:12:43.180098 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.180175 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 05:12:43.180527 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:41533: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
I20260925 05:12:43.271790 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } attempt: 3
I20260925 05:12:43.273082 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.272604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50842)
0925 05:12:43.272856 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.272947 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:12:43.274240 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.273916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50858)
0925 05:12:43.274048 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.274113 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:12:43.274482 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:41533: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
W20260925 05:12:43.413805 31522 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:12:43.414181 31522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:43.414300 31522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:43.414441 31522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 05:12:43.417717 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } attempt: 4
I20260925 05:12:43.418962 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.418416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50866)
0925 05:12:43.418737 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.418825 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260925 05:12:43.419937 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.419605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50882)
0925 05:12:43.419753 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.419823 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:12:43.420140 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:41533: Network error: Client connection negotiation failed: client connection to 127.8.215.61:41533: connect: Connection refused (error 111)
W20260925 05:12:43.422416 31522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:43.422498 31522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:43.422641 31522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:43.422715 31522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:43.422822 31522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:43.432978 31522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36075,127.8.215.61:41533,127.8.215.60:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab
--principal=kudu/127.8.215.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.215.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:43.434299 31522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:43.435652 31522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:43.447271 31527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:43.447369 31528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:43.447391 31530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:43.448302 31522 server_base.cc:1034] running on GCE node
I20260925 05:12:43.449072 31522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:43.450417 31522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:43.451650 31522 hybrid_clock.cc:648] HybridClock initialized: now 1790313163451577 us; error 34 us; skew 500 ppm
I20260925 05:12:43.452026 31522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:43 dist-test-slave-wpdm krb5kdc[31435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:43.455629 31522 init.cc:377] Logged in from keytab as kudu/127.8.215.61@KRBTEST.COM (short username kudu)
I20260925 05:12:43.457283 31522 webserver.cc:460] Webserver started at http://127.8.215.61:33549/ using document root <none> and password file <none>
I20260925 05:12:43.458014 31522 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:43.458110 31522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:43.458453 31522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:43.461822 31522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data/instance:
uuid: "a29880e4a7e549d0b3bab76297b294dc"
format_stamp: "Formatted at 2026-09-25 05:12:43 on dist-test-slave-wpdm"
server_key: "e99c8e41e5989566102a10ba93b26dc4"
server_key_iv: "a8402c0112e30279eb47a2bac18d015b"
server_key_version: "encryptionkey@0"
I20260925 05:12:43.462656 31522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal/instance:
uuid: "a29880e4a7e549d0b3bab76297b294dc"
format_stamp: "Formatted at 2026-09-25 05:12:43 on dist-test-slave-wpdm"
server_key: "e99c8e41e5989566102a10ba93b26dc4"
server_key_iv: "a8402c0112e30279eb47a2bac18d015b"
server_key_version: "encryptionkey@0"
I20260925 05:12:43.468158 31522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 05:12:43.471916 31537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.473397 31522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:43.473613 31522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal
uuid: "a29880e4a7e549d0b3bab76297b294dc"
format_stamp: "Formatted at 2026-09-25 05:12:43 on dist-test-slave-wpdm"
server_key: "e99c8e41e5989566102a10ba93b26dc4"
server_key_iv: "a8402c0112e30279eb47a2bac18d015b"
server_key_version: "encryptionkey@0"
I20260925 05:12:43.473835 31522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:43.495661 31522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:43.500140 31522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:43.500432 31522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:43.520407 31522 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.61:41533
I20260925 05:12:43.520471 31588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.61:41533 every 8 connection(s)
I20260925 05:12:43.521763 31522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
I20260925 05:12:43.525976 31589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:43.528532 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31522
I20260925 05:12:43.528766 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-1/wal/instance
I20260925 05:12:43.529347 9052 external_mini_cluster.cc:1428] Setting key c3b6a46bcfb2bf4c3a003a90b99847ee
I20260925 05:12:43.536216 31589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:43 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.215.60@KRBTEST.COM; defaulting to no policy
I20260925 05:12:43.561370 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.540178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.8.215.62:36075)
0925 05:12:43.540409 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.540416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.541701 (+ 1285us) server_negotiation.cc:407] Connection header received
0925 05:12:43.543835 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.543844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.543936 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.544049 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.545823 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.546961 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.548644 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.549483 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.549624 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.553850 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.553859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.553867 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.553916 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.557985 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.558774 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.558782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.558787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.558943 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.559366 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.559373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.559379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.559827 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.559998 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.560974 (+ 976us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.561149 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 05:12:43.562793 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.540552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36075 (local address 127.0.0.1:41690)
0925 05:12:43.541484 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.541548 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:43.543559 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:43.544278 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:43.544287 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:43.545006 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:43.545626 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.545645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.547161 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.547165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.548484 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.548497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.549704 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.549709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.549931 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.550645 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:43.550668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:43.553544 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:43.558151 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.558158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.558172 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.558621 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.559062 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.559065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.559067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.559235 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.560026 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:43.560034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:43.560525 (+ 491us) client_negotiation.cc:769] Sending connection context
0925 05:12:43.560898 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 05:12:43.561355 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.60@KRBTEST.COM" created.
I20260925 05:12:43.564913 31589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:43 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:12:43.584316 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.565665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50890)
0925 05:12:43.568888 (+ 3223us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.568903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:43.569003 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:43.569603 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:43.569606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:43.569997 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:43.570134 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.570141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.572626 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.572630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.573565 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.573576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.574975 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.574982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.575214 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.575881 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:43.575910 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:43.578698 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:43.581893 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.581901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.581906 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.582243 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.582614 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.582618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.582621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.582756 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.583358 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:43.583363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:43.583503 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 05:12:43.583635 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 05:12:43.583905 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3122}
I20260925 05:12:43.584359 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.565983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.8.215.61:41533)
0925 05:12:43.566790 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.566802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.569003 (+ 2201us) server_negotiation.cc:407] Connection header received
0925 05:12:43.569114 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.569126 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.569431 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.569622 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.570320 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.572390 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.573718 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.574817 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.574933 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.578903 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.578914 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.578927 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.578958 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.581753 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.582375 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.582389 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.582393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.582495 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.582877 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.582881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.582884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.583215 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.583434 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.583793 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.583998 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":371,"threads_started":1}
I20260925 05:12:43.587177 31589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:43.588455 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.588013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49924)
0925 05:12:43.588192 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.588301 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:12:43.588821 31540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.590422 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.590099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49928)
0925 05:12:43.590243 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.590322 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:12:43.590649 31589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab.
Entry for principal kudu/127.8.215.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab.
I20260925 05:12:43.600351 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:45285
--webserver_interface=127.8.215.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab
--principal=kudu/127.8.215.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:36075,127.8.215.61:41533,127.8.215.60:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:43.639961 31589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 1
I20260925 05:12:43.641270 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.640662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49940)
0925 05:12:43.640926 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.641054 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 05:12:43.642266 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.641947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49946)
0925 05:12:43.642092 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.642163 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:12:43.642486 31589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.710741 31589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 2
I20260925 05:12:43.712090 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.711595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49950)
0925 05:12:43.711821 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.711937 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 05:12:43.713517 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.713077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49962)
0925 05:12:43.713240 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.713362 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 05:12:43.713835 31589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.719415 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } attempt: 5
Sep 25 05:12:43 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:12:43.735374 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.720234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.8.215.61:41533)
0925 05:12:43.720425 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.720430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.720450 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:12:43.720719 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.720845 (+ 126us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.720958 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.721053 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.722053 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.722932 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.724633 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.725212 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.725315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.728877 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.728881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.728885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.728918 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.732036 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.732926 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.732935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.732939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.733093 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.733588 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.733596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.733599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.733952 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.734129 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.734665 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.734931 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:12:43.735373 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.720153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50896)
0925 05:12:43.720363 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.720384 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:43.720515 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:43.721134 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:43.721139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:43.721660 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:43.721907 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.721919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.723088 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.723094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.724449 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.724475 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.725372 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.725378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.725572 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.726236 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:43.726258 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:43.728622 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:43.732203 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.732210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.732215 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.732709 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.733260 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.733266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.733271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.733436 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.734119 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:43.734126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:43.734321 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 05:12:43.734549 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 05:12:43.734897 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:12:43.736789 31515 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:43.737902 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.737474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49978)
0925 05:12:43.737644 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.737726 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 05:12:43.738970 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.738616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49988)
0925 05:12:43.738763 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.738872 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 05:12:43.739233 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.790494 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 1
I20260925 05:12:43.791728 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.791253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49990)
0925 05:12:43.791473 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.791573 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 05:12:43.792788 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.792394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:49998)
0925 05:12:43.792552 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.792623 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:12:43.792992 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.805064 31589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 3
I20260925 05:12:43.806267 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.805782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50012)
0925 05:12:43.806027 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.806113 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 05:12:43.807341 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.806949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50026)
0925 05:12:43.807168 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.807236 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260925 05:12:43.807539 31589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.861258 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 2
I20260925 05:12:43.862677 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.862091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50032)
0925 05:12:43.862390 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.862495 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 05:12:43.863817 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.863417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50034)
0925 05:12:43.863598 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.863696 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:12:43.864076 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.950762 31589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 4
I20260925 05:12:43.951910 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.951415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50038)
0925 05:12:43.951651 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.951732 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 05:12:43.952354 31596 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:12:43.952826 31596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:43.952951 31596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260925 05:12:43.953032 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.952646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50040)
0925 05:12:43.952804 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.952881 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:12:43.953104 31596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:43.953284 31589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
W20260925 05:12:43.960707 31596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:43.960852 31596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:43.960947 31596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:43.961045 31596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:43.961133 31596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:43.970319 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 3
I20260925 05:12:43.971401 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.970918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50048)
0925 05:12:43.971149 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.971239 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:12:43.971907 31596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:36075,127.8.215.61:41533,127.8.215.60:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab
--principal=kudu/127.8.215.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:45285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.215.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:43.972579 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.972028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50062)
0925 05:12:43.972356 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.972446 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
W20260925 05:12:43.972878 31515 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:45285: Network error: Client connection negotiation failed: client connection to 127.8.215.60:45285: connect: Connection refused (error 111)
I20260925 05:12:43.973567 31596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:43.974882 31596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:43.986797 31601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:43.986827 31602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:43.986889 31604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:43.988248 31596 server_base.cc:1034] running on GCE node
I20260925 05:12:43.989176 31596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:43.990712 31596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:43.991972 31596 hybrid_clock.cc:648] HybridClock initialized: now 1790313163991886 us; error 67 us; skew 500 ppm
I20260925 05:12:43.992322 31596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:43 dist-test-slave-wpdm krb5kdc[31435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:43.995603 31596 init.cc:377] Logged in from keytab as kudu/127.8.215.60@KRBTEST.COM (short username kudu)
I20260925 05:12:43.997267 31596 webserver.cc:460] Webserver started at http://127.8.215.60:42349/ using document root <none> and password file <none>
I20260925 05:12:43.997987 31596 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:43.998119 31596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:43.998441 31596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:44.001510 31596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data/instance:
uuid: "94f0d6a8a0aa491f923b024adaced867"
format_stamp: "Formatted at 2026-09-25 05:12:43 on dist-test-slave-wpdm"
server_key: "dad4c3efe4af6091f8bef610d5ff541b"
server_key_iv: "63f296255acd14b88bdca3d22e409e6c"
server_key_version: "encryptionkey@0"
I20260925 05:12:44.002326 31596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal/instance:
uuid: "94f0d6a8a0aa491f923b024adaced867"
format_stamp: "Formatted at 2026-09-25 05:12:43 on dist-test-slave-wpdm"
server_key: "dad4c3efe4af6091f8bef610d5ff541b"
server_key_iv: "63f296255acd14b88bdca3d22e409e6c"
server_key_version: "encryptionkey@0"
I20260925 05:12:44.007813 31596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:12:44.011407 31611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:44.012861 31596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:12:44.013054 31596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal
uuid: "94f0d6a8a0aa491f923b024adaced867"
format_stamp: "Formatted at 2026-09-25 05:12:43 on dist-test-slave-wpdm"
server_key: "dad4c3efe4af6091f8bef610d5ff541b"
server_key_iv: "63f296255acd14b88bdca3d22e409e6c"
server_key_version: "encryptionkey@0"
I20260925 05:12:44.013374 31596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:44.039986 31596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:44.044090 31596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:44.044373 31596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:44.063079 31596 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.60:45285
I20260925 05:12:44.063099 31662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.60:45285 every 8 connection(s)
I20260925 05:12:44.064487 31596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
I20260925 05:12:44.067847 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31596
I20260925 05:12:44.068032 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312801912195-9052-0/minicluster-data/master-2/wal/instance
I20260925 05:12:44.068411 31663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:44.068542 9052 external_mini_cluster.cc:1428] Setting key f0fee9c5ce854abbd294dc3affd57e31
I20260925 05:12:44.078039 31663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:44.095347 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.070427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.8.215.60:45285)
0925 05:12:44.071308 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.071320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.071373 (+ 53us) server_negotiation.cc:407] Connection header received
0925 05:12:44.073213 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.073234 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.073657 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.074032 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.075869 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.077270 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.078870 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.079684 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.079890 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.087233 (+ 7343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.087243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.087256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.087304 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.090530 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.091348 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.091355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.091361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.091573 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.091937 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.091941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.091945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.092679 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.093009 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.093394 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.093884 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":417,"threads_started":1}
I20260925 05:12:44.098001 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.080693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.8.215.62:36075)
0925 05:12:44.081564 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.081570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.081998 (+ 428us) server_negotiation.cc:407] Connection header received
0925 05:12:44.082214 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.082239 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.082334 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.082463 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.084870 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.085621 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.087018 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.087713 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.087874 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.091565 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.091571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.091576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.091619 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.095198 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.095981 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.095986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.095989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.096147 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.096496 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.096501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.096503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.096881 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.096992 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.097488 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.097648 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":448,"threads_started":1}
I20260925 05:12:44.098222 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.080707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:36075 (local address 127.0.0.1:41696)
0925 05:12:44.081815 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.081890 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.082067 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.082526 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.082532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.083374 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:44.083796 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.083810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.085775 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.085782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.086872 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.086884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.087921 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.087926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.088178 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.088849 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:44.088866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:44.091370 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:44.095337 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.095346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.095351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.095861 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.096254 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.096257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.096259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.096388 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.097008 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:44.097017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:44.097214 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.097422 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.097735 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":665,"threads_started":1}
I20260925 05:12:44.099538 31663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:44.113320 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.100119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:41533 (local address 127.0.0.1:50900)
0925 05:12:44.100298 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.100318 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.100411 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.100742 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.100750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.101089 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:44.101220 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.101227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.102295 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.102300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.103519 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.103530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.104310 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.104317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.104500 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.105046 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:44.105062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:44.107155 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:44.110516 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.110522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.110528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.111035 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.111452 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.111455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.111457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.111660 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.112312 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:44.112317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:44.112477 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.112628 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.112957 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 05:12:44.113729 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.098843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.8.215.62:36075)
0925 05:12:44.098988 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.098994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.099039 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:12:44.099218 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.099224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.099304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.099403 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.100801 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.101648 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.103197 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.103606 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.103742 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.108019 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.108024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.108027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.108056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.110756 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.111561 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.111566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.111569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.111714 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.112099 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.112103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.112105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.112470 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.112761 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.113150 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.113395 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:12:44.113716 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.100288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.8.215.61:41533)
0925 05:12:44.100459 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.100463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.100492 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:12:44.100540 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.100543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.100598 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.100667 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.101360 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.102160 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.103651 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.104148 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.104279 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.107338 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.107343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.107346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.107378 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.110363 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.111153 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.111158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.111162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.111334 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.111800 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.111805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.111808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.112188 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.112305 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.112769 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.113011 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:12:44.114648 31663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:12:44.130345 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.115713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.8.215.60:45285)
0925 05:12:44.115869 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.115874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.116333 (+ 459us) server_negotiation.cc:407] Connection header received
0925 05:12:44.116511 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.116520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.116568 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.116687 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.117432 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.118529 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.120302 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.120964 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.121116 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.124832 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.124836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.124839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.124867 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.127690 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.128418 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.128422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.128424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.128528 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.128944 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.128947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.128949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.129208 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.129313 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.129730 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.129975 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:12:44.130590 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.115532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50074)
0925 05:12:44.116245 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.116259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.116384 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.116700 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.116704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.117131 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:44.117265 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.117273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.118694 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.118706 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.120132 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.120146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.121162 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.121169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.121403 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.122067 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:44.122089 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:44.124607 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:44.127820 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.127825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.127829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.128298 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.128633 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.128640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.128646 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.128839 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.129307 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:44.129313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:44.129477 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.129662 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.130144 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":168}
I20260925 05:12:44.131924 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.115510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.8.215.61:41533)
0925 05:12:44.115891 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.115897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.115913 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:12:44.115966 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.115969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.116021 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.116095 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.117574 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.118589 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.120311 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.121028 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.121135 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.126085 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.126089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.126092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.126122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.129095 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.129893 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.129899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.129903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.130060 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.130520 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.130526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.130532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.130864 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.131055 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.131431 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.131655 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260925 05:12:44.136543 31663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: Bootstrap starting.
I20260925 05:12:44.136972 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:44.139835 31663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:44.140923 31663 log.cc:824] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:44.139470 31490 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cda77d14988845e79c49d35a332818bd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41706
I20260925 05:12:44.143637 31663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: No bootstrap required, opened a new log
I20260925 05:12:44.150130 31515 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 4
I20260925 05:12:44.150924 31663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.151346 31663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f0d6a8a0aa491f923b024adaced867, State: Initialized, Role: FOLLOWER
I20260925 05:12:44.151878 31663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.152949 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.153373 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.154186 31663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
I20260925 05:12:44.168217 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.150818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50082)
0925 05:12:44.151018 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.151040 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.151162 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.151713 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.151717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.152080 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:44.152320 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.152346 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.153912 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.153920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.154996 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.155008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.156284 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.156290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.156545 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.157257 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:44.157285 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:44.159967 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:44.164953 (+ 4986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.164958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.164962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.165472 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.165950 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.165954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.165960 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.166174 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.166862 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:44.166867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:44.167076 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.167240 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.167645 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 05:12:44.169965 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.151122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.8.215.60:45285)
0925 05:12:44.151395 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.151403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.151428 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:12:44.151489 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.151494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.151567 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.151675 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.152537 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.153738 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.155149 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.156088 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.156248 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.160244 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.160254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.160262 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.160316 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.164684 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.165632 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.165639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.165644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.165798 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.166323 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.166327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.166330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.166703 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.168896 (+ 2193us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.169097 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.169405 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 05:12:44.175770 31681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:12:44.175916 31681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:12:44.176266 31515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: Bootstrap starting.
I20260925 05:12:44.167166 31596 master_runner.cc:186] Error getting master registration for 127.8.215.62:36075: OK, instance_id { permanent_uuid: "cda77d14988845e79c49d35a332818bd" instance_seqno: 1790313163008040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:12:44.178511 31596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:12:44.180235 31515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:44.181643 31515 log.cc:824] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:44.184270 31515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: No bootstrap required, opened a new log
I20260925 05:12:44.189378 31515 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.189702 31515 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda77d14988845e79c49d35a332818bd, State: Initialized, Role: FOLLOWER
I20260925 05:12:44.190099 31515 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.191041 31684 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.191465 31684 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.192185 31515 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:44.192373 31490 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:12:44.192634 31490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:44.196386 31490 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.199398 31638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cda77d14988845e79c49d35a332818bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94f0d6a8a0aa491f923b024adaced867"
I20260925 05:12:44.199926 31638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:44.203692 31638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda77d14988845e79c49d35a332818bd in term 1.
I20260925 05:12:44.204496 31466 leader_election.cc:302] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f0d6a8a0aa491f923b024adaced867, cda77d14988845e79c49d35a332818bd; no voters:
I20260925 05:12:44.205050 31684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:44.205132 31490 leader_election.cc:288] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [CANDIDATE]: Term 1 election: Requested vote from peers a29880e4a7e549d0b3bab76297b294dc (127.8.215.61:41533), 94f0d6a8a0aa491f923b024adaced867 (127.8.215.60:45285)
I20260925 05:12:44.219944 31684 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [term 1 LEADER]: Becoming Leader. State: Replica: cda77d14988845e79c49d35a332818bd, State: Running, Role: LEADER
I20260925 05:12:44.220837 31684 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.233865 31692 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda77d14988845e79c49d35a332818bd. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.234290 31692 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:44.251088 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.212046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.8.215.60:45285)
0925 05:12:44.212228 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.212234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.226803 (+ 14569us) server_negotiation.cc:407] Connection header received
0925 05:12:44.227274 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.227280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.227378 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.227461 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.229867 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.230966 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.233411 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.234156 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.234260 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.238014 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.238021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.238026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.238067 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.241755 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.246424 (+ 4669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.246431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.246434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.246742 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.247525 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.247531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.247559 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.248295 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.248769 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.249872 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.250298 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 05:12:44.251652 31469 tablet.cc:2369] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:44.252492 31589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } attempt: 5
W20260925 05:12:44.255666 31704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:12:44.255805 31704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:12:44.255928 31705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:44.257567 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.210041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.8.215.61:41533)
0925 05:12:44.210628 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.210638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.223319 (+ 12681us) server_negotiation.cc:407] Connection header received
0925 05:12:44.223701 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.223709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.223821 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.223920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.229441 (+ 5521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.230660 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.235485 (+ 4825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.236383 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.236508 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.247186 (+ 10678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.247193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.247200 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.247251 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.250923 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.253501 (+ 2578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.253509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.253514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.253774 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.254501 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.254507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.254510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.255127 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.255517 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.256493 (+ 976us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.256928 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260925 05:12:44.258291 31705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:44.262686 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.210790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.8.215.62:36075)
0925 05:12:44.211054 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.211062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.222058 (+ 10996us) server_negotiation.cc:407] Connection header received
0925 05:12:44.223428 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.223435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.223523 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.223598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.236862 (+ 13264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.238293 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.246427 (+ 8134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.247377 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.247503 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.254723 (+ 7220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.254730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.254734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.254794 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.258827 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.259719 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.259725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.259730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.259928 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.260342 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.260349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.260353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.260743 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.261072 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.261688 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.262254 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 05:12:44 dist-test-slave-wpdm krb5kdc[31435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
I20260925 05:12:44.246380 31448 master_runner.cc:186] Error getting master registration for 127.8.215.61:41533: OK, instance_id { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" instance_seqno: 1790313163514625 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:12:44.277118 31448 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:12:44.287045 31638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [term 1 FOLLOWER]: Refusing update from remote peer cda77d14988845e79c49d35a332818bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:44.288549 31692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:44.293146 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.253283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:45285 (local address 127.0.0.1:50104)
0925 05:12:44.254662 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.254684 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.254785 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.256184 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.256189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.256698 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:44.256948 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.256961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.263091 (+ 6130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.263097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.264533 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.264548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.265772 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.265778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.265997 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.266583 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:44.266608 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:44.269369 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:44.285474 (+ 16105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.285480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.285484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.286082 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.289770 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:44.289776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:44.289779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:44.289988 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:44.291771 (+ 1783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:44.291777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:44.292026 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.292214 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.292625 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246}
I20260925 05:12:44.295214 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.255200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.8.215.60:45285)
0925 05:12:44.255673 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.255679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.255698 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:12:44.255746 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.255752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.256018 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.257098 (+ 1080us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.257509 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.262599 (+ 5090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.264700 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.265577 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.265727 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.269584 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.269592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.269600 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.269647 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.285302 (+ 15655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.289103 (+ 3801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.289109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.289113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.289594 (+ 481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.290899 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.290906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.290909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.291524 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.291726 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.292429 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.292996 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
W20260925 05:12:44.296443 31617 tablet.cc:2369] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:44.300971 31589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: Bootstrap starting.
I20260925 05:12:44.303512 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda77d14988845e79c49d35a332818bd. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.303812 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.305099 31589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:44.306509 31589 log.cc:824] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:44.310151 31589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: No bootstrap required, opened a new log
I20260925 05:12:44.323065 31589 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.323555 31589 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29880e4a7e549d0b3bab76297b294dc, State: Initialized, Role: FOLLOWER
I20260925 05:12:44.323477 31684 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.323810 31684 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:44.324122 31589 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } }
I20260925 05:12:44.325567 31713 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.326401 31713 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.326782 31589 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:44.326869 31564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cda77d14988845e79c49d35a332818bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a29880e4a7e549d0b3bab76297b294dc"
I20260925 05:12:44.327310 31564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:44.331648 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.331893 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.332784 31705 catalog_manager.cc:1347] Generated new cluster ID: 5cc421b4b8644c39a79657069565f53d
I20260925 05:12:44.332871 31705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:44.335907 31564 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda77d14988845e79c49d35a332818bd in term 1.
I20260925 05:12:44.336923 31563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [term 1 FOLLOWER]: Refusing update from remote peer cda77d14988845e79c49d35a332818bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:44.339140 31692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:44.378252 31713 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda77d14988845e79c49d35a332818bd. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.378738 31713 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.379501 31719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.380065 31719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:44.388173 31684 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda77d14988845e79c49d35a332818bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda77d14988845e79c49d35a332818bd" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 36075 } } peers { permanent_uuid: "a29880e4a7e549d0b3bab76297b294dc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 41533 } } peers { permanent_uuid: "94f0d6a8a0aa491f923b024adaced867" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 45285 } } }
I20260925 05:12:44.388568 31684 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:44.393877 31705 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:44.401216 31705 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:44.427686 31729 catalog_manager.cc:1259] Loaded cluster ID: 5cc421b4b8644c39a79657069565f53d
I20260925 05:12:44.427802 31729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: loading cluster ID for follower catalog manager: success
I20260925 05:12:44.438995 31729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: acquiring CA information for follower catalog manager: success
I20260925 05:12:44.440266 31705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda77d14988845e79c49d35a332818bd: Generated new TSK 0
W20260925 05:12:44.440457 31729 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 05:12:44.441439 31705 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:12:44.599843 31667 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.564415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.8.215.62:36075)
0925 05:12:44.564682 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.564689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.595752 (+ 31063us) server_negotiation.cc:407] Connection header received
0925 05:12:44.598072 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.598078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.598169 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.598285 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.599594 (+ 1309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41724: BlockingRecv error: recv got EOF from 127.0.0.1:41724 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 05:12:44.602072 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.570986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.8.215.61:41533)
0925 05:12:44.571223 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.571231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.597770 (+ 26539us) server_negotiation.cc:407] Connection header received
0925 05:12:44.600532 (+ 2762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.600538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.600647 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.600741 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.601825 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50924: BlockingRecv error: recv got EOF from 127.0.0.1:50924 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 05:12:44.605088 31666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.580848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.8.215.60:45285)
0925 05:12:44.581067 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.581075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.602868 (+ 21793us) server_negotiation.cc:407] Connection header received
0925 05:12:44.603742 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.603748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.603832 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.603910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.604836 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50108: BlockingRecv error: recv got EOF from 127.0.0.1:50108 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 05:12:44.648258 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.634691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.8.215.61:41533)
0925 05:12:44.634892 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.634898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.645919 (+ 11021us) server_negotiation.cc:407] Connection header received
0925 05:12:44.646537 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.646544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.646681 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.646763 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.647939 (+ 1176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50936: BlockingRecv error: recv got EOF from 127.0.0.1:50936 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 05:12:44.708858 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.698999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.8.215.61:41533)
0925 05:12:44.699262 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.699271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.706903 (+ 7632us) server_negotiation.cc:407] Connection header received
0925 05:12:44.707259 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.707266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.707385 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.707471 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.708583 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50938: BlockingRecv error: recv got EOF from 127.0.0.1:50938 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 05:12:44.760885 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.747051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.8.215.61:41533)
0925 05:12:44.747252 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.747258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.758577 (+ 11319us) server_negotiation.cc:407] Connection header received
0925 05:12:44.759156 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.759163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.759268 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.759369 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.760578 (+ 1209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50952: BlockingRecv error: recv got EOF from 127.0.0.1:50952 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 05:12:44.815331 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.800242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.8.215.61:41533)
0925 05:12:44.800501 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.800509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.813490 (+ 12981us) server_negotiation.cc:407] Connection header received
0925 05:12:44.813799 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.813805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.813920 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.813991 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.815079 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50954: BlockingRecv error: recv got EOF from 127.0.0.1:50954 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260925 05:12:44.871558 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.858654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.8.215.61:41533)
0925 05:12:44.858872 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.858878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.869603 (+ 10725us) server_negotiation.cc:407] Connection header received
0925 05:12:44.870115 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.870122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.870220 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.870290 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.871289 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50970: BlockingRecv error: recv got EOF from 127.0.0.1:50970 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 05:12:44.896217 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.893982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.8.215.61:41533)
0925 05:12:44.894224 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.894233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.894775 (+ 542us) server_negotiation.cc:407] Connection header received
0925 05:12:44.894998 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.895010 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.895081 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.895201 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.895965 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50978: BlockingRecv error: recv got EOF from 127.0.0.1:50978 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 05:12:44.934303 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:44.931856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.8.215.61:41533)
0925 05:12:44.932090 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.932100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.932665 (+ 565us) server_negotiation.cc:407] Connection header received
0925 05:12:44.933040 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.933045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.933126 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.933241 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.934043 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50988: BlockingRecv error: recv got EOF from 127.0.0.1:50988 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 05:12:45.006906 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:45.004764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.8.215.61:41533)
0925 05:12:45.005052 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.005061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.005524 (+ 463us) server_negotiation.cc:407] Connection header received
0925 05:12:45.005688 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.005695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.005825 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.005949 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:45.006643 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51000: BlockingRecv error: recv got EOF from 127.0.0.1:51000 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260925 05:12:45.143812 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:45.142084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.8.215.61:41533)
0925 05:12:45.142264 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.142268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.142639 (+ 371us) server_negotiation.cc:407] Connection header received
0925 05:12:45.142798 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.142802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.142874 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.142970 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:45.143618 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51004: BlockingRecv error: recv got EOF from 127.0.0.1:51004 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:12:45.178299 31681 catalog_manager.cc:1259] Loaded cluster ID: 5cc421b4b8644c39a79657069565f53d
I20260925 05:12:45.178448 31681 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: loading cluster ID for follower catalog manager: success
I20260925 05:12:45.180985 31681 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: acquiring CA information for follower catalog manager: success
I20260925 05:12:45.182684 31681 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94f0d6a8a0aa491f923b024adaced867: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:12:45.405052 31593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:45.403324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.8.215.61:41533)
0925 05:12:45.403512 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.403517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.403794 (+ 277us) server_negotiation.cc:407] Connection header received
0925 05:12:45.403967 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.403971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.404043 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.404137 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:45.404841 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51008: BlockingRecv error: recv got EOF from 127.0.0.1:51008 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:12:45.443038 31729 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a29880e4a7e549d0b3bab76297b294dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:12:45.927502 31796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:45.922047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.8.215.61:41533)
0925 05:12:45.922520 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.922524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.922648 (+ 124us) server_negotiation.cc:407] Connection header received
0925 05:12:45.922810 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.922814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.922973 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.923089 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:45.923476 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.924191 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.925458 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.925846 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.925946 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:45.926345 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:45.926942 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:45.927058 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:45.927195 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 05:12:45.927260 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260925 05:12:45.938807 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:45.933298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.8.215.60:45285)
0925 05:12:45.933809 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.933830 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.933847 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:12:45.933964 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.933967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.934116 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.934250 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:45.934627 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.935425 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.936697 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.937103 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.937195 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:45.937641 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:45.938188 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:45.938295 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:45.938390 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:12:45.938465 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
I20260925 05:12:45.952687 31796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:45.945004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.8.215.61:41533)
0925 05:12:45.945217 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.945226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.946737 (+ 1511us) server_negotiation.cc:407] Connection header received
0925 05:12:45.946965 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.946972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.947062 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.947159 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:45.947681 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.948711 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.950496 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.951192 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.951364 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:45.951865 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:45.952016 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:45.952120 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:45.952297 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:12:45.952385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:12:45.954041 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:45.945126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.8.215.62:36075)
0925 05:12:45.946066 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.946072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.946094 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:12:45.946357 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.946366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.946581 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.947323 (+ 742us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:45.947499 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.948616 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.951209 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.951868 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.951997 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:45.952605 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:45.953066 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:45.953191 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:45.953610 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 05:12:45.953705 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":660,"threads_started":1}
I20260925 05:12:45.955747 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:45.945447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.8.215.60:45285)
0925 05:12:45.945647 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:45.945654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:45.947621 (+ 1967us) server_negotiation.cc:407] Connection header received
0925 05:12:45.948982 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:45.948988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:45.949081 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:45.949175 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:45.950438 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.951181 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.953371 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:45.954125 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:45.954271 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:45.955011 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:45.955146 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:45.955263 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:45.955358 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:12:45.955443 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:12:45.959357 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31448
I20260925 05:12:45.960397 31510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:46.071302 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31448
I20260925 05:12:46.094317 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31522
I20260925 05:12:46.095376 31584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:46.200783 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31522
I20260925 05:12:46.221712 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31596
I20260925 05:12:46.222832 31658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:46.330454 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31596
2026-09-25T05:12:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5160 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:46 dist-test-slave-wpdm krb5kdc[31816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:46 dist-test-slave-wpdm krb5kdc[31816](info): set up 2 sockets
Sep 25 05:12:46 dist-test-slave-wpdm krb5kdc[31816](info): commencing operation
krb5kdc: starting...
W20260925 05:12:47.458292 9052 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/test-user.keytab.
Sep 25 05:12:47 dist-test-slave-wpdm krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.215.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
Entry for principal kudu/127.8.215.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab.
I20260925 05:12:47.732275 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:38235
--webserver_interface=127.8.215.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:38235,127.8.215.61:35977,127.8.215.60:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:48.060987 31829 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:12:48.061285 31829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:48.061358 31829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:48.061439 31829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:48.068783 31829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:48.068868 31829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:48.068930 31829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:48.068981 31829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:48.069046 31829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:48.078764 31829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:38235,127.8.215.61:35977,127.8.215.60:36929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.62.keytab
--principal=kudu/127.8.215.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.62:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.215.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:48.079980 31829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:48.081244 31829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:48.093176 31834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:48.093142 31835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:48.093250 31837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:48.094179 31829 server_base.cc:1034] running on GCE node
I20260925 05:12:48.094914 31829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:48.096611 31829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:48.097954 31829 hybrid_clock.cc:648] HybridClock initialized: now 1790313168097844 us; error 59 us; skew 500 ppm
I20260925 05:12:48.098383 31829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:48 dist-test-slave-wpdm krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:48.101600 31829 init.cc:377] Logged in from keytab as kudu/127.8.215.62@KRBTEST.COM (short username kudu)
I20260925 05:12:48.102937 31829 webserver.cc:460] Webserver started at http://127.8.215.62:39401/ using document root <none> and password file <none>
I20260925 05:12:48.103662 31829 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:48.103797 31829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:48.104100 31829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:48.107033 31829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data/instance:
uuid: "e3b3194cd66b40768910fa9405321a3a"
format_stamp: "Formatted at 2026-09-25 05:12:48 on dist-test-slave-wpdm"
server_key: "ce1d9dbf27aa418b4f833566ea4e9272"
server_key_iv: "1225abe00b2bbd5fd85e9180745984b2"
server_key_version: "encryptionkey@0"
I20260925 05:12:48.107833 31829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal/instance:
uuid: "e3b3194cd66b40768910fa9405321a3a"
format_stamp: "Formatted at 2026-09-25 05:12:48 on dist-test-slave-wpdm"
server_key: "ce1d9dbf27aa418b4f833566ea4e9272"
server_key_iv: "1225abe00b2bbd5fd85e9180745984b2"
server_key_version: "encryptionkey@0"
I20260925 05:12:48.112998 31829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:12:48.116430 31844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:48.117784 31829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:48.117964 31829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal
uuid: "e3b3194cd66b40768910fa9405321a3a"
format_stamp: "Formatted at 2026-09-25 05:12:48 on dist-test-slave-wpdm"
server_key: "ce1d9dbf27aa418b4f833566ea4e9272"
server_key_iv: "1225abe00b2bbd5fd85e9180745984b2"
server_key_version: "encryptionkey@0"
I20260925 05:12:48.118178 31829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:48.133809 31829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:48.138266 31829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:48.138578 31829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:48.157316 31829 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.62:38235
I20260925 05:12:48.157343 31895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.62:38235 every 8 connection(s)
I20260925 05:12:48.158802 31829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:48.162432 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:48.168933 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31829
I20260925 05:12:48.169127 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-0/wal/instance
I20260925 05:12:48.169525 9052 external_mini_cluster.cc:1428] Setting key e437b7950d806ba165a91f4cc064b858
I20260925 05:12:48.169180 31896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:48 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:48.194511 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.173409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.8.215.62:38235)
0925 05:12:48.174030 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:48.174036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:48.174078 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:12:48.176415 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:48.176424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:48.176709 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:48.176934 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:48.178576 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.180225 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.182466 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.183037 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.183171 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.187251 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:48.187259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:48.187272 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:48.187311 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:48.190138 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.190842 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.190855 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.190857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.190999 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.191377 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.191382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.191385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.191869 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:48.192086 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:48.192601 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 05:12:48.193037 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":461,"thread_start_us":304,"threads_started":1}
I20260925 05:12:48.194537 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.172285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38235 (local address 127.0.0.1:43460)
0925 05:12:48.173172 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.173242 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:48.176000 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:48.176985 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:48.176995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:48.177859 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:48.178353 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.178380 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.180380 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.180388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.182323 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.182341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.183283 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.183293 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.183533 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.184237 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:48.184288 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:48.187037 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:48.190290 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.190302 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.190315 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.190706 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.191109 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.191115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.191119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.191265 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.192005 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:48.192017 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:48.192230 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 05:12:48.192473 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 05:12:48.193151 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":178,"thread_start_us":367,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.61@KRBTEST.COM; defaulting to no policy
I20260925 05:12:48.197950 31896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:48.198979 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.198555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50452)
0925 05:12:48.198710 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.198813 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.61@KRBTEST.COM" created.
W20260925 05:12:48.199308 31847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
I20260925 05:12:48.200985 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.200487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50460)
0925 05:12:48.200672 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.200794 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 05:12:48.201207 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:35977: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab.
Entry for principal kudu/127.8.215.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab.
I20260925 05:12:48.232197 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:35977
--webserver_interface=127.8.215.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab
--principal=kudu/127.8.215.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:38235,127.8.215.61:35977,127.8.215.60:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:48.250563 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } attempt: 1
I20260925 05:12:48.251761 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.251322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50462)
0925 05:12:48.251519 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.251632 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 05:12:48.252844 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.252423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50472)
0925 05:12:48.252566 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.252634 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:12:48.253072 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:35977: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
I20260925 05:12:48.321388 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } attempt: 2
I20260925 05:12:48.322677 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.322174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50480)
0925 05:12:48.322365 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.322485 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:12:48.323755 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.323379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50494)
0925 05:12:48.323542 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.323635 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 05:12:48.324012 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:35977: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
I20260925 05:12:48.415266 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } attempt: 3
I20260925 05:12:48.416543 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.416107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50510)
0925 05:12:48.416315 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.416408 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 05:12:48.417639 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.417275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50516)
0925 05:12:48.417444 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.417515 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 05:12:48.417825 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:35977: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
I20260925 05:12:48.561046 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } attempt: 4
I20260925 05:12:48.562199 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.561678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50532)
0925 05:12:48.561988 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.562073 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 05:12:48.563189 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.562835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50544)
0925 05:12:48.563001 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.563074 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:12:48.563369 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.61:35977: Network error: Client connection negotiation failed: client connection to 127.8.215.61:35977: connect: Connection refused (error 111)
W20260925 05:12:48.564265 31903 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:12:48.564615 31903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:48.564783 31903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:48.564942 31903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:48.572480 31903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:48.572607 31903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:48.572778 31903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:48.572860 31903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:48.572949 31903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:48.582942 31903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:38235,127.8.215.61:35977,127.8.215.60:36929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.61.keytab
--principal=kudu/127.8.215.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.61:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.215.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:48.584261 31903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:48.585634 31903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:48.597162 31908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:48.597210 31909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:48.597322 31911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:48.597772 31903 server_base.cc:1034] running on GCE node
I20260925 05:12:48.598570 31903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:48.599942 31903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:48.601186 31903 hybrid_clock.cc:648] HybridClock initialized: now 1790313168601122 us; error 56 us; skew 500 ppm
I20260925 05:12:48.601504 31903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:48 dist-test-slave-wpdm krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:48.604449 31903 init.cc:377] Logged in from keytab as kudu/127.8.215.61@KRBTEST.COM (short username kudu)
I20260925 05:12:48.605840 31903 webserver.cc:460] Webserver started at http://127.8.215.61:35537/ using document root <none> and password file <none>
I20260925 05:12:48.606531 31903 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:48.606664 31903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:48.606992 31903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:48.609992 31903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data/instance:
uuid: "fd18d9ab91314d839304bd7a28b7d3fc"
format_stamp: "Formatted at 2026-09-25 05:12:48 on dist-test-slave-wpdm"
server_key: "5f5c5431648633b5cc54de404f7c0951"
server_key_iv: "8d450191e611e46f6f77c4acfbc019d4"
server_key_version: "encryptionkey@0"
I20260925 05:12:48.610762 31903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal/instance:
uuid: "fd18d9ab91314d839304bd7a28b7d3fc"
format_stamp: "Formatted at 2026-09-25 05:12:48 on dist-test-slave-wpdm"
server_key: "5f5c5431648633b5cc54de404f7c0951"
server_key_iv: "8d450191e611e46f6f77c4acfbc019d4"
server_key_version: "encryptionkey@0"
I20260925 05:12:48.615789 31903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 05:12:48.619310 31918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:48.620667 31903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:12:48.620898 31903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal
uuid: "fd18d9ab91314d839304bd7a28b7d3fc"
format_stamp: "Formatted at 2026-09-25 05:12:48 on dist-test-slave-wpdm"
server_key: "5f5c5431648633b5cc54de404f7c0951"
server_key_iv: "8d450191e611e46f6f77c4acfbc019d4"
server_key_version: "encryptionkey@0"
I20260925 05:12:48.621125 31903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:48.647105 31903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:48.651451 31903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:48.651731 31903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:48.670822 31903 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.61:35977
I20260925 05:12:48.670879 31969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.61:35977 every 8 connection(s)
I20260925 05:12:48.672132 31903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/data/info.pb
I20260925 05:12:48.675889 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:48.680567 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31903
I20260925 05:12:48.680786 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-1/wal/instance
I20260925 05:12:48.681334 9052 external_mini_cluster.cc:1428] Setting key 75767e1b4eac199fe67ef46a6556237b
I20260925 05:12:48.683074 31970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:48 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:48.706092 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.686344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.8.215.62:38235)
0925 05:12:48.686574 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:48.686580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:48.687828 (+ 1248us) server_negotiation.cc:407] Connection header received
0925 05:12:48.689765 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:48.689773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:48.689869 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:48.689994 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:48.691372 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.692493 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.694973 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.695416 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.695520 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.699851 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:48.699857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:48.699860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:48.699898 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:48.702775 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.703581 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.703593 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.703595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.703744 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.704220 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.704225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.704227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.704517 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:48.704688 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:48.705466 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 05:12:48.705706 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:12:48.707098 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.686837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38235 (local address 127.0.0.1:43470)
0925 05:12:48.687651 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.687695 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:48.689490 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:48.690172 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:48.690179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:48.690675 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:48.691204 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.691219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.692651 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.692658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.694806 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.694823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.695640 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.695650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.695990 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.696787 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:48.696821 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:48.699588 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:48.702942 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.702960 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.702978 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.703438 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.703875 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.703880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.703884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.704080 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.704660 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:48.704668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:48.705077 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 05:12:48.705355 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 05:12:48.705912 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":350,"threads_started":1}
WARNING: no policy specified for kudu/127.8.215.60@KRBTEST.COM; defaulting to no policy
I20260925 05:12:48.708986 31970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.215.60@KRBTEST.COM" created.
Sep 25 05:12:48 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:12:48.725931 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.709625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50560)
0925 05:12:48.709790 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.709815 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:48.709984 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:48.711102 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:48.711106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:48.711582 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:48.711766 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.711778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.713258 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.713267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.714876 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.714895 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.715719 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.715733 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.715960 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.716582 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:48.716612 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:48.719117 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:48.723042 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.723049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.723053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.723496 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.723918 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.723924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.723928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.724092 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.724880 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:48.724885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:48.725026 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 05:12:48.725183 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 05:12:48.725472 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:12:48.726006 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.709957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.8.215.61:35977)
0925 05:12:48.710604 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:48.710610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:48.710634 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:12:48.710673 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:48.710678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:48.710922 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:48.711039 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:48.711953 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.713096 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.715029 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.715557 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.715655 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.719329 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:48.719340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:48.719352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:48.719382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:48.722764 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.723655 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.723664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.723668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.723802 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.724242 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.724247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.724251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.724638 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:48.724893 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:48.725363 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 05:12:48.725586 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":365,"threads_started":1}
I20260925 05:12:48.728560 31970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:48.729652 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.729218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39830)
0925 05:12:48.729396 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.729518 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 05:12:48.729887 31921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
I20260925 05:12:48.731451 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.731133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39832)
0925 05:12:48.731284 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.731359 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:12:48.731648 31970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.215.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab.
Entry for principal kudu/127.8.215.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab.
I20260925 05:12:48.748153 9052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
/tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:36929
--webserver_interface=127.8.215.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab
--principal=kudu/127.8.215.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.215.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.215.62:38235,127.8.215.61:35977,127.8.215.60:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:48.780939 31970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 1
I20260925 05:12:48.782094 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.781631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39834)
0925 05:12:48.781821 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.781947 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 05:12:48.783167 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.782827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39842)
0925 05:12:48.782992 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.783069 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:12:48.783425 31970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
I20260925 05:12:48.851692 31970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 2
I20260925 05:12:48.852902 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.852373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39844)
0925 05:12:48.852580 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.852683 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 05:12:48.853931 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.853584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39848)
0925 05:12:48.853731 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.853816 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:12:48.854121 31970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
I20260925 05:12:48.862620 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } attempt: 5
Sep 25 05:12:48 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:12:48.877502 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.863369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.8.215.61:35977)
0925 05:12:48.863549 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:48.863554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:48.863625 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:12:48.863825 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:48.863828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:48.863908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:48.863988 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:48.865093 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.865819 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.867531 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.867992 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.868100 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.871842 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:48.871847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:48.871850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:48.871883 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:48.874689 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.875582 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.875585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.875588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.875722 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:48.876151 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:48.876154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:48.876156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:48.876412 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:48.876522 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:48.877102 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 05:12:48.877282 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:12:48.877663 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.863303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50570)
0925 05:12:48.863520 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.863542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:48.863686 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:48.864059 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:48.864066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:48.864658 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:48.864926 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.864939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.865963 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.865968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.867354 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:48.867369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:48.868165 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:48.868171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:48.868353 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:48.869056 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:48.869079 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:48.871615 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:48.874840 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.874847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.874854 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.875463 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.875832 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:48.875840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:48.875844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:48.876037 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:48.876543 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:48.876551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:48.876791 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 05:12:48.876983 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 05:12:48.877307 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:12:48.878841 31896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:48.879948 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.879508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39854)
0925 05:12:48.879688 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.879779 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 05:12:48.881023 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.880633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39866)
0925 05:12:48.880809 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.880908 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 05:12:48.881237 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
I20260925 05:12:48.932477 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 1
I20260925 05:12:48.933831 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.933173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39876)
0925 05:12:48.933446 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.933584 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 05:12:48.934868 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.934509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39888)
0925 05:12:48.934670 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.934752 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:12:48.935032 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
I20260925 05:12:48.945349 31970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 3
I20260925 05:12:48.946604 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.945994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39904)
0925 05:12:48.946284 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.946403 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 05:12:48.948043 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:48.947455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39920)
0925 05:12:48.947731 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:48.947839 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260925 05:12:48.948315 31970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
I20260925 05:12:49.003260 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 2
I20260925 05:12:49.004524 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.004003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39922)
0925 05:12:49.004248 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.004346 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 05:12:49.005627 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.005310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39936)
0925 05:12:49.005459 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.005528 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:12:49.005843 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
W20260925 05:12:49.087631 31977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:12:49.087956 31977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:49.088106 31977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:49.088254 31977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 05:12:49.091562 31970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 4
I20260925 05:12:49.092765 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.092178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39950)
0925 05:12:49.092426 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.092540 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 05:12:49.093807 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.093422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39952)
0925 05:12:49.093593 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.093679 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 05:12:49.094024 31970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
W20260925 05:12:49.096338 31977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:49.096412 31977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:49.096547 31977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:49.096618 31977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:49.096745 31977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:49.108484 31977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.215.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.215.62:38235,127.8.215.61:35977,127.8.215.60:36929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/krb5kdc/kudu_127.8.215.60.keytab
--principal=kudu/127.8.215.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.215.60:36929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.215.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:12:49.109833 31977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:49.111155 31977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:12:49.112056 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 3
I20260925 05:12:49.113160 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.112681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39954)
0925 05:12:49.112921 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.113010 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 05:12:49.114114 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.113762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39964)
0925 05:12:49.113958 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.114019 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 05:12:49.114317 31896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.215.60:36929: Network error: Client connection negotiation failed: client connection to 127.8.215.60:36929: connect: Connection refused (error 111)
W20260925 05:12:49.122813 31983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:49.123000 31982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:49.123163 31985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:49.124125 31977 server_base.cc:1034] running on GCE node
I20260925 05:12:49.125005 31977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:49.126478 31977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:49.127725 31977 hybrid_clock.cc:648] HybridClock initialized: now 1790313169127653 us; error 66 us; skew 500 ppm
I20260925 05:12:49.128087 31977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:49.135836 31977 init.cc:377] Logged in from keytab as kudu/127.8.215.60@KRBTEST.COM (short username kudu)
I20260925 05:12:49.137387 31977 webserver.cc:460] Webserver started at http://127.8.215.60:43623/ using document root <none> and password file <none>
I20260925 05:12:49.138085 31977 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:49.138200 31977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:49.138542 31977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:49.141513 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data/instance:
uuid: "9d64ed33b10d4edfab96fcff35ec092d"
format_stamp: "Formatted at 2026-09-25 05:12:49 on dist-test-slave-wpdm"
server_key: "2f0b0ec60f8a089d3994acd5df3fd3ff"
server_key_iv: "babf32c8f74fa25d3c20f1b592d1b848"
server_key_version: "encryptionkey@0"
I20260925 05:12:49.142279 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal/instance:
uuid: "9d64ed33b10d4edfab96fcff35ec092d"
format_stamp: "Formatted at 2026-09-25 05:12:49 on dist-test-slave-wpdm"
server_key: "2f0b0ec60f8a089d3994acd5df3fd3ff"
server_key_iv: "babf32c8f74fa25d3c20f1b592d1b848"
server_key_version: "encryptionkey@0"
I20260925 05:12:49.147325 31977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:12:49.150777 31992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:49.152287 31977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:12:49.152487 31977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data,/tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal
uuid: "9d64ed33b10d4edfab96fcff35ec092d"
format_stamp: "Formatted at 2026-09-25 05:12:49 on dist-test-slave-wpdm"
server_key: "2f0b0ec60f8a089d3994acd5df3fd3ff"
server_key_iv: "babf32c8f74fa25d3c20f1b592d1b848"
server_key_version: "encryptionkey@0"
I20260925 05:12:49.152710 31977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:49.171099 31977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:49.175567 31977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:49.175853 31977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:49.193701 31977 rpc_server.cc:307] RPC server started. Bound to: 127.8.215.60:36929
I20260925 05:12:49.193719 32043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.215.60:36929 every 8 connection(s)
I20260925 05:12:49.195041 31977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/data/info.pb
I20260925 05:12:49.198920 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:49.199589 9052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu as pid 31977
I20260925 05:12:49.199766 9052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjaqqHJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312801912195-9052-0/minicluster-data/master-2/wal/instance
I20260925 05:12:49.200227 9052 external_mini_cluster.cc:1428] Setting key 052124ec25a022b713be86fff515f9d5
I20260925 05:12:49.208818 32044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:49.225075 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.202073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.8.215.60:36929)
0925 05:12:49.202887 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.202902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.202954 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:12:49.205693 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.205722 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.206270 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.206729 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.208083 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.210125 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.211404 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.212566 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.212769 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.217165 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.217172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.217182 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.217219 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.220635 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.221544 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.221553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.221562 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.221768 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.222175 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.222180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.222184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.222805 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.223112 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.223527 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.223938 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":325,"threads_started":1}
I20260925 05:12:49.227500 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.211691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.8.215.62:38235)
0925 05:12:49.212263 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.212271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.212998 (+ 727us) server_negotiation.cc:407] Connection header received
0925 05:12:49.213068 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.213072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.213133 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.213218 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.214713 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.215460 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.216850 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.217361 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.217455 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.221273 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.221277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.221281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.221321 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.224424 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.225328 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.225334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.225337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.225513 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.225961 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.225965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.225967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.226208 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.226327 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.226846 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.227085 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
I20260925 05:12:49.227509 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.211701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.62:38235 (local address 127.0.0.1:43474)
0925 05:12:49.212624 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.212687 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:49.212859 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:49.213328 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:49.213333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:49.214168 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:49.214540 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.214557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.215621 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.215629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.216664 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.216676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.217549 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.217556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.217781 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.218308 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:49.218349 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:49.221059 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:49.224615 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.224623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.224628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.225081 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.225646 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.225651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.225654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.225843 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.226337 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:49.226344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:49.226523 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 05:12:49.226713 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 05:12:49.227093 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":552,"threads_started":1}
I20260925 05:12:49.229187 32044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.62@KRBTEST.COM
I20260925 05:12:49.243391 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.229833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.61:35977 (local address 127.0.0.1:50584)
0925 05:12:49.229979 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.229997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:49.230077 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:49.230438 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:49.230442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:49.230800 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:49.230937 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.230944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.231911 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.231917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.233427 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.233461 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.234178 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.234184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.234376 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.234872 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:49.234887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:49.237067 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:49.240985 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.240989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.240991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.241361 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.241735 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.241741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.241747 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.241927 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.242559 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:49.242566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:49.242712 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 05:12:49.242819 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:12:49.243045 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:12:49.243531 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.229958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.8.215.61:35977)
0925 05:12:49.230139 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.230144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.230163 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:12:49.230225 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.230233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.230311 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.230417 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.231114 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.231772 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.233586 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.234027 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.234166 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.237237 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.237241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.237245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.237276 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.240855 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.241491 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.241495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.241498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.241619 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.242069 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.242073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.242077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.242435 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.242572 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.242949 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.243174 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:12:49.244117 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.228429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.8.215.62:38235)
0925 05:12:49.228565 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.228570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.228585 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:12:49.228629 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.228634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.228713 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.228835 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.230254 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.230903 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.232205 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.232901 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.233024 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.237929 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.237938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.237943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.237981 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.241278 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.242070 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.242075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.242078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.242226 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.242609 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.242614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.242617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.242949 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.243204 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.243544 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.243790 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:12:49.244856 32044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.60@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.215.61@KRBTEST.COM
I20260925 05:12:49.259058 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.245489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39980)
0925 05:12:49.245622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.245636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:49.245747 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:49.246130 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:49.246137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:49.246623 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:49.246753 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.246761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.248072 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.248077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.249345 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.249357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.250297 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.250302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.250450 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.251017 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:49.251040 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:49.253137 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:49.256537 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.256540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.256543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.257048 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.257426 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.257433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.257438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.257567 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.258062 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:49.258066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:49.258206 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 05:12:49.258362 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 05:12:49.258656 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":4}
I20260925 05:12:49.259104 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.245704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.8.215.60:36929)
0925 05:12:49.245865 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.245871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.245887 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:12:49.245933 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.245938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.246005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.246102 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.246904 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.247953 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.249470 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.250142 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.250282 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.253317 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.253324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.253330 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.253372 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.256396 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.257158 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.257165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.257171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.257314 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.257675 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.257679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.257683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.257950 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.258097 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.258452 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.258805 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":53}
I20260925 05:12:49.261251 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.245944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.8.215.61:35977)
0925 05:12:49.246101 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.246106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.246121 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:12:49.246216 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.246221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.246294 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.246393 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.247854 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.248970 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.250544 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.251167 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.251295 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.255599 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.255605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.255610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.255647 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.258556 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.259326 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.259333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.259338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.259505 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.259883 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.259889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.259892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.260138 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.260439 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.260697 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.260948 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:12:49.264622 32044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: Bootstrap starting.
I20260925 05:12:49.265897 9052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:49.268151 32044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:49.269248 32044 log.cc:824] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:49.268198 31871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e3b3194cd66b40768910fa9405321a3a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478
I20260925 05:12:49.271970 32044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: No bootstrap required, opened a new log
I20260925 05:12:49.278879 32044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.279199 32044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64ed33b10d4edfab96fcff35ec092d, State: Initialized, Role: FOLLOWER
I20260925 05:12:49.279646 32044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.280486 32051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.280815 32051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.281595 32044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:12:49.290879 32062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:12:49.291026 32062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:12:49.291627 31896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 4
I20260925 05:12:49.288575 31977 master_runner.cc:186] Error getting master registration for 127.8.215.62:38235: OK, instance_id { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" instance_seqno: 1790313168152024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:12:49.300097 31977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.62@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
I20260925 05:12:49.312075 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.292214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:39986)
0925 05:12:49.292482 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.292502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:49.292611 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:49.296714 (+ 4103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:49.296780 (+ 66us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:49.297345 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:49.297552 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.297564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.299066 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.299072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.300457 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.300469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.301785 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.301791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.302004 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.302731 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:49.302756 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:49.305250 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:49.309301 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.309306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.309311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.309801 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.310219 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.310224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.310228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.310390 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.311048 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:49.311052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:49.311238 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 05:12:49.311381 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 05:12:49.311666 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 05:12:49.312107 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.295974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.8.215.60:36929)
0925 05:12:49.296218 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.296224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.296251 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:12:49.296320 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.296329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.296410 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.296517 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.297722 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.298838 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.300599 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.301592 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.301767 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.305453 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.305474 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.305481 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.305528 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.309166 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.309930 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.309936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.309940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.310096 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.310509 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.310516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.310522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.310923 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.311067 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.311516 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.311741 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:12:49.317047 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a: Bootstrap starting.
I20260925 05:12:49.320339 31896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:49.321574 31896 log.cc:824] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:49.324703 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a: No bootstrap required, opened a new log
I20260925 05:12:49.330107 31896 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.330463 31896 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b3194cd66b40768910fa9405321a3a, State: Initialized, Role: FOLLOWER
I20260925 05:12:49.331002 31896 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.332254 32065 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.332929 32065 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.333528 31896 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:49.333621 31871 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:12:49.333808 31871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:49.338129 31871 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.340206 31871 leader_election.cc:288] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [CANDIDATE]: Term 1 election: Requested vote from peers fd18d9ab91314d839304bd7a28b7d3fc (127.8.215.61:35977), 9d64ed33b10d4edfab96fcff35ec092d (127.8.215.60:36929)
I20260925 05:12:49.341224 32019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3b3194cd66b40768910fa9405321a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9d64ed33b10d4edfab96fcff35ec092d"
I20260925 05:12:49.341739 32019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:49.346834 32019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b3194cd66b40768910fa9405321a3a in term 1.
I20260925 05:12:49.347795 31847 leader_election.cc:302] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64ed33b10d4edfab96fcff35ec092d, e3b3194cd66b40768910fa9405321a3a; no voters:
I20260925 05:12:49.348444 32065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:49.355885 32065 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [term 1 LEADER]: Becoming Leader. State: Replica: e3b3194cd66b40768910fa9405321a3a, State: Running, Role: LEADER
I20260925 05:12:49.356910 32065 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.363336 32078 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b3194cd66b40768910fa9405321a3a. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.363898 32078 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:49.367864 32083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 05:12:49.391479 31850 tablet.cc:2369] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:49.393198 31970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } attempt: 5
I20260925 05:12:49.403249 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.381000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.8.215.61:35977)
0925 05:12:49.381188 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.381194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.386192 (+ 4998us) server_negotiation.cc:407] Connection header received
0925 05:12:49.386338 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.386344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.386422 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.386517 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.388646 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.389776 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.391637 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.392290 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.392381 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.396407 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.396414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.396418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.396456 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.399996 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.400850 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.400856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.400859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.401013 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.401453 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.401459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.401463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.401828 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.402079 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.402642 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.402872 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:12:49.411554 32083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:49.421450 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.386779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.8.215.60:36929)
0925 05:12:49.386969 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.386975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.404190 (+ 17215us) server_negotiation.cc:407] Connection header received
0925 05:12:49.404447 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.404454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.404535 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.404602 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.406446 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.407685 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.409582 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.410285 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.410382 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.413515 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.413522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.413527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.413570 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.418077 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.418939 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.418946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.418950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.419118 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.419586 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.419592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.419596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.419968 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.420222 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.420804 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.421041 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 05:12:49 dist-test-slave-wpdm krb5kdc[31816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.215.61@KRBTEST.COM for kudu/127.8.215.60@KRBTEST.COM
I20260925 05:12:49.425264 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.373087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.8.215.62:38235)
0925 05:12:49.373385 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.373391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.380470 (+ 7079us) server_negotiation.cc:407] Connection header received
0925 05:12:49.381987 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.381994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.382089 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.382166 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.384389 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.385611 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.391034 (+ 5423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.391737 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.391852 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.414752 (+ 22900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.414759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.414766 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.414809 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.419191 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.421519 (+ 2328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.421525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.421528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.422090 (+ 562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.423345 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.423352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.423355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.423723 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.423954 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.424540 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.424817 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:12:49.381278 31829 master_runner.cc:186] Error getting master registration for 127.8.215.61:35977: OK, instance_id { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" instance_seqno: 1790313168665342 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260925 05:12:49.427999 31998 tablet.cc:2369] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:49.433952 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.393871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.215.60:36929 (local address 127.0.0.1:40008)
0925 05:12:49.394214 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:49.394233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:49.394329 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:49.412591 (+ 18262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:49.412597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:49.413108 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:49.413304 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.413316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.416924 (+ 3608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.416930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.418319 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:49.418334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.419999 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.420006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:49.420228 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.420818 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:49.420840 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:49.423627 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:49.430353 (+ 6726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.430371 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.430374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.430935 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.431648 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:49.431654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:49.431658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:49.431825 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:49.432856 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:49.432861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:49.433029 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 05:12:49.433196 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 05:12:49.433515 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260925 05:12:49.434434 31829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:12:49.435750 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.394635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.8.215.60:36929)
0925 05:12:49.411821 (+ 17186us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.411828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.411855 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:12:49.411907 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.411913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.411989 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.413476 (+ 1487us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.413808 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.416181 (+ 2373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.418587 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.419768 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.420550 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.424045 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.424053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.424058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.424167 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.430092 (+ 5925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.431174 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.431183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.431186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.431455 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.432017 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.432023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.432026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.432632 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.434501 (+ 1869us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.434708 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.435055 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17001,"thread_start_us":17422,"threads_started":1}
I20260925 05:12:49.441826 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: Bootstrap starting.
I20260925 05:12:49.446394 32019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [term 1 FOLLOWER]: Refusing update from remote peer e3b3194cd66b40768910fa9405321a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:49.446836 31970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:49.447520 32078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:49.448099 31970 log.cc:824] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:49.465188 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: No bootstrap required, opened a new log
I20260925 05:12:49.476096 32051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b3194cd66b40768910fa9405321a3a. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.476444 32051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.481423 32051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.481719 32051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.481935 32091 mvcc.cc:205] Tried to move back new op lower bound from 7333122742042124288 to 7333122741696385024. Current Snapshot: MvccSnapshot[applied={T|T < 7333122742042124288}]
I20260925 05:12:49.483455 31970 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.483852 31970 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd18d9ab91314d839304bd7a28b7d3fc, State: Initialized, Role: FOLLOWER
I20260925 05:12:49.484381 31970 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } }
I20260925 05:12:49.490960 32083 catalog_manager.cc:1347] Generated new cluster ID: 3d58860986ca44198fdce2d19448e652
I20260925 05:12:49.491099 32083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:49.494382 32078 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.494659 32078 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:49.499797 32095 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.500236 32095 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.501158 31970 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:49.502197 31945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3b3194cd66b40768910fa9405321a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd18d9ab91314d839304bd7a28b7d3fc"
I20260925 05:12:49.504911 31945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:49.508574 31945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b3194cd66b40768910fa9405321a3a in term 1.
I20260925 05:12:49.509301 31944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [term 1 FOLLOWER]: Refusing update from remote peer e3b3194cd66b40768910fa9405321a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:49.510950 32065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:49.547315 32083 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:49.551721 32083 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:49.562153 32078 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.562481 32078 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:49.565471 32095 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b3194cd66b40768910fa9405321a3a. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.565846 32095 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.565456 32103 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3b3194cd66b40768910fa9405321a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b3194cd66b40768910fa9405321a3a" member_type: VOTER last_known_addr { host: "127.8.215.62" port: 38235 } } peers { permanent_uuid: "fd18d9ab91314d839304bd7a28b7d3fc" member_type: VOTER last_known_addr { host: "127.8.215.61" port: 35977 } } peers { permanent_uuid: "9d64ed33b10d4edfab96fcff35ec092d" member_type: VOTER last_known_addr { host: "127.8.215.60" port: 36929 } } }
I20260925 05:12:49.567492 32103 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:49.586371 32083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b3194cd66b40768910fa9405321a3a: Generated new TSK 0
I20260925 05:12:49.587627 32083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:49.602489 32111 catalog_manager.cc:1259] Loaded cluster ID: 3d58860986ca44198fdce2d19448e652
I20260925 05:12:49.602663 32111 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: loading cluster ID for follower catalog manager: success
I20260925 05:12:49.618993 32111 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: acquiring CA information for follower catalog manager: success
I20260925 05:12:49.622287 32111 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd18d9ab91314d839304bd7a28b7d3fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:12:49.744648 32048 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.732390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.8.215.62:38235)
0925 05:12:49.732608 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.732614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.741499 (+ 8885us) server_negotiation.cc:407] Connection header received
0925 05:12:49.743007 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.743014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.743106 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.743185 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.744385 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43498: BlockingRecv error: recv got EOF from 127.0.0.1:43498 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 05:12:49.748162 31974 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.742120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.8.215.61:35977)
0925 05:12:49.742318 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.742324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.746083 (+ 3759us) server_negotiation.cc:407] Connection header received
0925 05:12:49.746786 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.746792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.746873 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.746951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.747925 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50610: BlockingRecv error: recv got EOF from 127.0.0.1:50610 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 05:12:49.753348 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.750934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.8.215.60:36929)
0925 05:12:49.751138 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.751145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.751621 (+ 476us) server_negotiation.cc:407] Connection header received
0925 05:12:49.751849 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.751855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.751957 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.752031 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.753101 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40022: BlockingRecv error: recv got EOF from 127.0.0.1:40022 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:12:49.802901 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.787494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.8.215.61:35977)
0925 05:12:49.787675 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.787681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.795047 (+ 7366us) server_negotiation.cc:407] Connection header received
0925 05:12:49.795567 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.795573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.795808 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.795944 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:49.796624 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.797839 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.799873 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.800539 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.800632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.801367 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:49.802091 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:49.802204 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.802456 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.802559 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 05:12:49.847512 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.837608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.8.215.60:36929)
0925 05:12:49.837810 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.837819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.845317 (+ 7498us) server_negotiation.cc:407] Connection header received
0925 05:12:49.845882 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.845890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.846028 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.846116 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.847221 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40026: BlockingRecv error: recv got EOF from 127.0.0.1:40026 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:12:49.893954 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.878714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.8.215.61:35977)
0925 05:12:49.878923 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.878934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.886228 (+ 7294us) server_negotiation.cc:407] Connection header received
0925 05:12:49.886725 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.886731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.886832 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.886898 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:49.887590 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.888832 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.891314 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.891990 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.892083 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.892707 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:49.892891 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:49.892960 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.893535 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.893624 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 05:12:49.923276 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.916408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.8.215.60:36929)
0925 05:12:49.916589 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.916595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.921315 (+ 4720us) server_negotiation.cc:407] Connection header received
0925 05:12:49.921604 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.921611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.921728 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.921871 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.923047 (+ 1176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40036: BlockingRecv error: recv got EOF from 127.0.0.1:40036 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:12:49.938323 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.930771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.8.215.61:35977)
0925 05:12:49.930952 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.930958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.931492 (+ 534us) server_negotiation.cc:407] Connection header received
0925 05:12:49.931689 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.931696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.931838 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.931957 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:49.932477 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.933809 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.935949 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.936701 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.936900 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.937380 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:49.937527 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:49.937630 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.937862 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.937969 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:12:49.952702 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.950342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.8.215.60:36929)
0925 05:12:49.950524 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.950532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.951096 (+ 564us) server_negotiation.cc:407] Connection header received
0925 05:12:49.951290 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.951300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.951418 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.951550 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.952435 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40046: BlockingRecv error: recv got EOF from 127.0.0.1:40046 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:12:49.971314 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.963823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.8.215.61:35977)
0925 05:12:49.964007 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.964014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.964521 (+ 507us) server_negotiation.cc:407] Connection header received
0925 05:12:49.964709 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.964716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.964881 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.965002 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:49.965469 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.966752 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.969019 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.969782 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.969933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.970445 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:49.970587 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:49.970690 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.970872 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.970971 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 05:12:49.982967 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:49.978874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.8.215.60:36929)
0925 05:12:49.979129 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.979136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.981300 (+ 2164us) server_negotiation.cc:407] Connection header received
0925 05:12:49.981503 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.981509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.981610 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.981706 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.982670 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40052: BlockingRecv error: recv got EOF from 127.0.0.1:40052 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 05:12:50.006161 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.998689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.8.215.61:35977)
0925 05:12:49.998882 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.998889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.999430 (+ 541us) server_negotiation.cc:407] Connection header received
0925 05:12:49.999628 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.999634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.999727 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.999883 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.000323 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.001652 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.003753 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.004561 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.004697 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.005221 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.005393 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.005496 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.005665 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.005752 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 05:12:50.016459 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:50.014319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.8.215.60:36929)
0925 05:12:50.014572 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.014581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.015120 (+ 539us) server_negotiation.cc:407] Connection header received
0925 05:12:50.015319 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.015325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.015406 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.015507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:50.016243 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40062: BlockingRecv error: recv got EOF from 127.0.0.1:40062 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:12:50.041518 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.035628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.8.215.61:35977)
0925 05:12:50.035850 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.035860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.036396 (+ 536us) server_negotiation.cc:407] Connection header received
0925 05:12:50.036623 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.036627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.036696 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.036826 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.037240 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.038149 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.039673 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.040150 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.040281 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.040714 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.040870 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.040967 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.041116 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.041205 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 05:12:50.053151 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:50.050765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.8.215.60:36929)
0925 05:12:50.051054 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.051063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.051625 (+ 562us) server_negotiation.cc:407] Connection header received
0925 05:12:50.051837 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.051842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.051928 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.052038 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:50.052907 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40066: BlockingRecv error: recv got EOF from 127.0.0.1:40066 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 05:12:50.094465 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.088363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.8.215.61:35977)
0925 05:12:50.088564 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.088571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.088909 (+ 338us) server_negotiation.cc:407] Connection header received
0925 05:12:50.089132 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.089139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.089245 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.089371 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.089749 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.091052 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.092405 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.093155 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.093306 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.093668 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.093809 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.093913 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.094013 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.094096 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 05:12:50.102054 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:50.100160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.8.215.60:36929)
0925 05:12:50.100432 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.100439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.100652 (+ 213us) server_negotiation.cc:407] Connection header received
0925 05:12:50.100790 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.100797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.100903 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.101023 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:50.101797 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40078: BlockingRecv error: recv got EOF from 127.0.0.1:40078 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 05:12:50.183784 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.178338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.8.215.61:35977)
0925 05:12:50.178560 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.178567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.178897 (+ 330us) server_negotiation.cc:407] Connection header received
0925 05:12:50.179122 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.179131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.179205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.179317 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.179743 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.180609 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.182086 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.182571 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.182699 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.183078 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.183177 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.183273 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.183426 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.183505 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 05:12:50.191370 32088 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:50.189515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.8.215.60:36929)
0925 05:12:50.189715 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.189724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.190023 (+ 299us) server_negotiation.cc:407] Connection header received
0925 05:12:50.190234 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.190239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.190325 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.190444 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:50.191175 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40094: BlockingRecv error: recv got EOF from 127.0.0.1:40094 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:12:50.293057 32062 catalog_manager.cc:1259] Loaded cluster ID: 3d58860986ca44198fdce2d19448e652
I20260925 05:12:50.293147 32062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: loading cluster ID for follower catalog manager: success
I20260925 05:12:50.295620 32062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: acquiring CA information for follower catalog manager: success
I20260925 05:12:50.297444 32062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d64ed33b10d4edfab96fcff35ec092d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:12:50.328088 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.323013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.8.215.61:35977)
0925 05:12:50.323248 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.323256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.323471 (+ 215us) server_negotiation.cc:407] Connection header received
0925 05:12:50.323703 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.323714 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.323784 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.323889 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.324353 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.325125 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.326501 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.326893 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.327029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.327423 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.327535 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.327633 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.327749 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.327827 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:12:50.339854 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.333903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.8.215.60:36929)
0925 05:12:50.334108 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.334117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.334533 (+ 416us) server_negotiation.cc:407] Connection header received
0925 05:12:50.334740 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.334751 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.334917 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.335036 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.335471 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.336180 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.337565 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.337996 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.338135 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.338619 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.339222 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.339384 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.339493 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.339574 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:12:50.353072 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.345904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.8.215.62:38235)
0925 05:12:50.346444 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.346451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.346551 (+ 100us) server_negotiation.cc:407] Connection header received
0925 05:12:50.346872 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.346880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.347118 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.347302 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.348540 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.349466 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.350977 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.351378 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.351518 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.352020 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.352381 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.352510 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.352639 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.352753 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":292,"threads_started":1}
I20260925 05:12:50.354370 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.346792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.8.215.61:35977)
0925 05:12:50.347000 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.347007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.347167 (+ 160us) server_negotiation.cc:407] Connection header received
0925 05:12:50.347337 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.347343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.347438 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.347541 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.347999 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.348928 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.351683 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.352622 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.352775 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.353611 (+ 836us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.353757 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.353848 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.354044 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.354124 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:12:50.358566 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31829
I20260925 05:12:50.359112 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:50.347326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.8.215.60:36929)
0925 05:12:50.347568 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:50.347574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:50.353043 (+ 5469us) server_negotiation.cc:407] Connection header received
0925 05:12:50.353510 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:50.353517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:50.353610 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:50.353705 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:50.354227 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.355186 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.357402 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:50.357868 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:50.357998 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:50.358430 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:50.358532 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:50.358626 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:50.358791 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 05:12:50.358869 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:12:50.359508 31891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:50.469501 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31829
I20260925 05:12:50.491444 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31903
I20260925 05:12:50.492553 31965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:50.597931 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31903
I20260925 05:12:50.618772 9052 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu pid 31977
I20260925 05:12:50.619966 32039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:12:50.727047 9052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjaqqHJ/build/asan/bin/kudu with pid 31977
2026-09-25T05:12:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4391 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9551 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368765 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST